SherbyNet - /page/wikibook/entry.mu

<!DOCTYPE html>

<title>Expectation–maximization algorithm</title>

</head>

Expectation–maximization algorithm

</h1>

</header>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hlist dl,.mw-parser-output .hlist ol,.mw-parser-output .hlist ul{margin:0;padding:0}.mw-parser-output .hlist dd,.mw-parser-output .hlist dt,.mw-parser-output .hlist li{margin:0;display:inline}.mw-parser-output .hlist.inline,.mw-parser-output .hlist.inline dl,.mw-parser-output .hlist.inline ol,.mw-parser-output .hlist.inline ul,.mw-parser-output .hlist dl dl,.mw-parser-output .hlist dl ol,.mw-parser-output .hlist dl ul,.mw-parser-output .hlist ol dl,.mw-parser-output .hlist ol ol,.mw-parser-output .hlist ol ul,.mw-parser-output .hlist ul dl,.mw-parser-output .hlist ul ol,.mw-parser-output .hlist ul ul{display:inline}.mw-parser-output .hlist .mw-empty-li{display:none}.mw-parser-output .hlist dt::after{content:": "}.mw-parser-output .hlist dd::after,.mw-parser-output .hlist li::after{content:" · ";font-weight:bold}.mw-parser-output .hlist dd:last-child::after,.mw-parser-output .hlist dt:last-child::after,.mw-parser-output .hlist li:last-child::after{content:none}.mw-parser-output .hlist dd dd:first-child::before,.mw-parser-output .hlist dd dt:first-child::before,.mw-parser-output .hlist dd li:first-child::before,.mw-parser-output .hlist dt dd:first-child::before,.mw-parser-output .hlist dt dt:first-child::before,.mw-parser-output .hlist dt li:first-child::before,.mw-parser-output .hlist li dd:first-child::before,.mw-parser-output .hlist li dt:first-child::before,.mw-parser-output .hlist li li:first-child::before{content:" (";font-weight:normal}.mw-parser-output .hlist dd dd:last-child::after,.mw-parser-output .hlist dd dt:last-child::after,.mw-parser-output .hlist dd li:last-child::after,.mw-parser-output .hlist dt dd:last-child::after,.mw-parser-output .hlist dt dt:last-child::after,.mw-parser-output .hlist dt li:last-child::after,.mw-parser-output .hlist li dd:last-child::after,.mw-parser-output .hlist li dt:last-child::after,.mw-parser-output .hlist li li:last-child::after{content:")";font-weight:normal}.mw-parser-output .hlist ol{counter-reset:listitem}.mw-parser-output .hlist ol>li{counter-increment:listitem}.mw-parser-output .hlist ol>li::before{content:" "counter(listitem)"\a0 "}.mw-parser-output .hlist dd ol>li:first-child::before,.mw-parser-output .hlist dt ol>li:first-child::before,.mw-parser-output .hlist li ol>li:first-child::before{content:" ("counter(listitem)"\a0 "}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r1246091330">

/* start https://en.wikipedia.org/ */

.mw-parser-output .sidebar{width:22em;float:right;clear:right;margin:0.5em 0 1em 1em;background:var(--background-color-neutral-subtle,#f8f9fa);border:1px solid var(--border-color-base,#a2a9b1);padding:0.2em;text-align:center;line-height:1.4em;font-size:88%;border-collapse:collapse;display:table}body.skin-minerva .mw-parser-output .sidebar{display:table!important;float:right!important;margin:0.5em 0 1em 1em!important}.mw-parser-output .sidebar-subgroup{width:100%;margin:0;border-spacing:0}.mw-parser-output .sidebar-left{float:left;clear:left;margin:0.5em 1em 1em 0}.mw-parser-output .sidebar-none{float:none;clear:both;margin:0.5em 1em 1em 0}.mw-parser-output .sidebar-outer-title{padding:0 0.4em 0.2em;font-size:125%;line-height:1.2em;font-weight:bold}.mw-parser-output .sidebar-top-image{padding:0.4em}.mw-parser-output .sidebar-top-caption,.mw-parser-output .sidebar-pretitle-with-top-image,.mw-parser-output .sidebar-caption{padding:0.2em 0.4em 0;line-height:1.2em}.mw-parser-output .sidebar-pretitle{padding:0.4em 0.4em 0;line-height:1.2em}.mw-parser-output .sidebar-title,.mw-parser-output .sidebar-title-with-pretitle{padding:0.2em 0.8em;font-size:145%;line-height:1.2em}.mw-parser-output .sidebar-title-with-pretitle{padding:0.1em 0.4em}.mw-parser-output .sidebar-image{padding:0.2em 0.4em 0.4em}.mw-parser-output .sidebar-heading{padding:0.1em 0.4em}.mw-parser-output .sidebar-content{padding:0 0.5em 0.4em}.mw-parser-output .sidebar-content-with-subgroup{padding:0.1em 0.4em 0.2em}.mw-parser-output .sidebar-above,.mw-parser-output .sidebar-below{padding:0.3em 0.8em;font-weight:bold}.mw-parser-output .sidebar-collapse .sidebar-above,.mw-parser-output .sidebar-collapse .sidebar-below{border-top:1px solid #aaa;border-bottom:1px solid #aaa}.mw-parser-output .sidebar-navbar{text-align:right;font-size:115%;padding:0 0.4em 0.4em}.mw-parser-output .sidebar-list-title{padding:0 0.4em;text-align:left;font-weight:bold;line-height:1.6em;font-size:105%}.mw-parser-output .sidebar-list-title-c{padding:0 0.4em;text-align:center;margin:0 3.3em}@media(max-width:640px){body.mediawiki .mw-parser-output .sidebar{width:100%!important;clear:both;float:none!important;margin-left:0!important;margin-right:0!important}}body.skin--responsive .mw-parser-output .sidebar a>img{max-width:none!important}@media screen{html.skin-theme-clientpref-night .mw-parser-output .sidebar:not(.notheme) .sidebar-list-title,html.skin-theme-clientpref-night .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle{background:transparent!important}html.skin-theme-clientpref-night .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle a{color:var(--color-progressive)!important}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .sidebar:not(.notheme) .sidebar-list-title,html.skin-theme-clientpref-os .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle{background:transparent!important}html.skin-theme-clientpref-os .mw-parser-output .sidebar:not(.notheme) .sidebar-title-with-pretitle a{color:var(--color-progressive)!important}}@media print{body.ns-0 .mw-parser-output .sidebar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><style data-mw-deduplicate="TemplateStyles:r886047488">

/* start https://en.wikipedia.org/ */

.mw-parser-output .nobold{font-weight:normal}

/* end https://en.wikipedia.org/ */

</style><table class="sidebar sidebar-collapse nomobile nowraplinks"><tbody><tr><td class="sidebar-pretitle">Part of a series on</td></tr><tr><th class="sidebar-title-with-pretitle"><a href="Machine_learning" title="Machine learning">Machine learning</a> and <a href="Data_mining" title="Data mining">data mining</a></th></tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Paradigms</div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Supervised_learning" title="Supervised learning">Supervised learning</a></li>

<li><a href="Unsupervised_learning" title="Unsupervised learning">Unsupervised learning</a></li>

<li><a href="Semi-supervised_learning" class="mw-redirect" title="Semi-supervised learning">Semi-supervised learning</a></li>

<li><a href="Self-supervised_learning" title="Self-supervised learning">Self-supervised learning</a></li>

<li><a href="Reinforcement_learning" title="Reinforcement learning">Reinforcement learning</a></li>

<li><a href="Meta-learning_(computer_science)" title="Meta-learning (computer science)">Meta-learning</a></li>

<li><a href="Online_machine_learning" title="Online machine learning">Online learning</a></li>

<li><a href="Batch_learning" class="mw-redirect" title="Batch learning">Batch learning</a></li>

<li><a href="Curriculum_learning" title="Curriculum learning">Curriculum learning</a></li>

<li><a href="Rule-based_machine_learning" title="Rule-based machine learning">Rule-based learning</a></li>

<li><a href="Neuro-symbolic_AI" title="Neuro-symbolic AI">Neuro-symbolic AI</a></li>

<li><a href="Neuromorphic_engineering" class="mw-redirect" title="Neuromorphic engineering">Neuromorphic engineering</a></li>

<li><a href="Quantum_machine_learning" title="Quantum machine learning">Quantum machine learning</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Problems</div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Statistical_classification" title="Statistical classification">Classification</a></li>

<li><a href="Generative_model" title="Generative model">Generative modeling</a></li>

<li><a href="Regression_analysis" title="Regression analysis">Regression</a></li>

<li><a href="Cluster_analysis" title="Cluster analysis">Clustering</a></li>

<li><a href="Dimensionality_reduction" title="Dimensionality reduction">Dimensionality reduction</a></li>

<li><a href="Density_estimation" title="Density estimation">Density estimation</a></li>

<li><a href="Anomaly_detection" title="Anomaly detection">Anomaly detection</a></li>

<li><a href="Data_cleaning" class="mw-redirect" title="Data cleaning">Data cleaning</a></li>

<li><a href="Automated_machine_learning" title="Automated machine learning">AutoML</a></li>

<li><a href="Association_rule_learning" title="Association rule learning">Association rules</a></li>

<li><a href="Semantic_analysis_(machine_learning)" title="Semantic analysis (machine learning)">Semantic analysis</a></li>

<li><a href="Structured_prediction" title="Structured prediction">Structured prediction</a></li>

<li><a href="Feature_engineering" title="Feature engineering">Feature engineering</a></li>

<li><a href="Feature_learning" title="Feature learning">Feature learning</a></li>

<li><a href="Learning_to_rank" title="Learning to rank">Learning to rank</a></li>

<li><a href="Grammar_induction" title="Grammar induction">Grammar induction</a></li>

<li><a href="Ontology_learning" title="Ontology learning">Ontology learning</a></li>

<li><a href="Multimodal_learning" title="Multimodal learning">Multimodal learning</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><div style="display: inline-block; line-height: 1.2em; padding: .1em 0;"><a href="Supervised_learning" title="Supervised learning">Supervised learning</a> (<a href="Statistical_classification" title="Statistical classification">classification</a> • <a href="Regression_analysis" title="Regression analysis">regression</a>) </div></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Apprenticeship_learning" title="Apprenticeship learning">Apprenticeship learning</a></li>

<li><a href="Decision_tree_learning" title="Decision tree learning">Decision trees</a></li>

<li><a href="Ensemble_learning" title="Ensemble learning">Ensembles</a>

<ul><li><a href="Bootstrap_aggregating" title="Bootstrap aggregating">Bagging</a></li>

<li><a href="Boosting_(machine_learning)" title="Boosting (machine learning)">Boosting</a></li>

<li><a href="Random_forest" title="Random forest">Random forest</a></li></ul></li>

<li><a href="Linear_regression" title="Linear regression">Linear regression</a></li>

<li><a href="Naive_Bayes_classifier" title="Naive Bayes classifier">Naive Bayes</a></li>

<li><a href="Artificial_neural_network" class="mw-redirect" title="Artificial neural network">Artificial neural networks</a></li>

<li><a href="Logistic_regression" title="Logistic regression">Logistic regression</a></li>

<li><a href="Perceptron" title="Perceptron">Perceptron</a></li>

<li><a href="Relevance_vector_machine" title="Relevance vector machine">Relevance vector machine (RVM)</a></li>

<li><a href="Support_vector_machine" title="Support vector machine">Support vector machine (SVM)</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><a href="Cluster_analysis" title="Cluster analysis">Clustering</a></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="BIRCH" title="BIRCH">BIRCH</a></li>

<li><a href="Hierarchical_clustering" title="Hierarchical clustering">Hierarchical</a></li>

<li><a href="K-means_clustering" title="K-means clustering">k-means</a></li>

<li><a href="Fuzzy_clustering" title="Fuzzy clustering">Fuzzy</a></li>

<li> <a href="DBSCAN" title="DBSCAN">DBSCAN</a></li>

<li><a href="OPTICS_algorithm" title="OPTICS algorithm">OPTICS</a></li>

<li><a href="Mean_shift" title="Mean shift">Mean shift</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><a href="Dimensionality_reduction" title="Dimensionality reduction">Dimensionality reduction</a></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Factor_analysis" title="Factor analysis">Factor analysis</a></li>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><a href="Structured_prediction" title="Structured prediction">Structured prediction</a></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Graphical_model" title="Graphical model">Graphical models</a>

<ul><li><a href="Bayesian_network" title="Bayesian network">Bayes net</a></li>

<li><a href="Conditional_random_field" title="Conditional random field">Conditional random field</a></li>

<li><a href="Hidden_Markov_model" title="Hidden Markov model">Hidden Markov</a></li></ul></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><a href="Anomaly_detection" title="Anomaly detection">Anomaly detection</a></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Random_sample_consensus" title="Random sample consensus">RANSAC</a></li>

<li><a href="Local_outlier_factor" title="Local outlier factor">Local outlier factor</a></li>

<li><a href="Isolation_forest" title="Isolation forest">Isolation forest</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><a href="Neural_network_(machine_learning)" title="Neural network (machine learning)">Neural networks</a></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Autoencoder" title="Autoencoder">Autoencoder</a></li>

<li><a href="Deep_learning" title="Deep learning">Deep learning</a></li>

<li><a href="Feedforward_neural_network" title="Feedforward neural network">Feedforward neural network</a></li>

<li><a href="Recurrent_neural_network" title="Recurrent neural network">Recurrent neural network</a>

<li><a href="Reservoir_computing" title="Reservoir computing">reservoir computing</a></li></ul></li>

<li><a href="Boltzmann_machine" title="Boltzmann machine">Boltzmann machine</a>

<ul><li><a href="Restricted_Boltzmann_machine" title="Restricted Boltzmann machine">Restricted</a></li></ul></li>

<li><a href="Diffusion_model" title="Diffusion model">Diffusion model</a></li>

<li><a href="Convolutional_neural_network" title="Convolutional neural network">Convolutional neural network</a>

<li><a href="LeNet" title="LeNet">LeNet</a></li>

<li><a href="AlexNet" title="AlexNet">AlexNet</a></li>

<li><a href="DeepDream" title="DeepDream">DeepDream</a></li></ul></li>

<li><a href="Neural_field" title="Neural field">Neural field</a>

<ul><li><a href="Neural_radiance_field" title="Neural radiance field">Neural radiance field</a></li>

<li><a href="Physics-informed_neural_networks" title="Physics-informed neural networks">Physics-informed neural networks</a></li></ul></li>

<li><a href="Transformer_(deep_learning_architecture)" title="Transformer (deep learning architecture)">Transformer</a>

<ul><li><a href="Vision_transformer" title="Vision transformer">Vision</a></li></ul></li>

<li><a href="Mamba_(deep_learning_architecture)" title="Mamba (deep learning architecture)">Mamba</a></li>

<li><a href="Spiking_neural_network" title="Spiking neural network">Spiking neural network</a></li>

<li><a href="Memtransistor" title="Memtransistor">Memtransistor</a></li>

<li><a href="Electrochemical_RAM" title="Electrochemical RAM">Electrochemical RAM</a> (ECRAM)</li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)"><a href="Reinforcement_learning" title="Reinforcement learning">Reinforcement learning</a></div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Q-learning" title="Q-learning">Q-learning</a></li>

<li><a href="Policy_gradient_method" title="Policy gradient method">Policy gradient</a></li>

<li><a href="State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action" title="State–action–reward–state–action">SARSA</a></li>

<li><a href="Temporal_difference_learning" title="Temporal difference learning">Temporal difference (TD)</a></li>

<li><a href="Multi-agent_reinforcement_learning" title="Multi-agent reinforcement learning">Multi-agent</a>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Learning with humans</div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Active_learning_(machine_learning)" title="Active learning (machine learning)">Active learning</a></li>

<li><a href="Crowdsourcing" title="Crowdsourcing">Crowdsourcing</a></li>

<li><a href="Human-in-the-loop" title="Human-in-the-loop">Human-in-the-loop</a></li>

<li><a href="Mechanistic_interpretability" title="Mechanistic interpretability">Mechanistic interpretability</a></li>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Model diagnostics</div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Coefficient_of_determination" title="Coefficient of determination">Coefficient of determination</a></li>

<li><a href="Confusion_matrix" title="Confusion matrix">Confusion matrix</a></li>

<li><a href="Learning_curve_(machine_learning)" title="Learning curve (machine learning)">Learning curve</a></li>

<li><a href="Receiver_operating_characteristic" title="Receiver operating characteristic">ROC curve</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Mathematical foundations</div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Kernel_machines" class="mw-redirect" title="Kernel machines">Kernel machines</a></li>

<li><a href="Bias%E2%80%93variance_tradeoff" title="Bias–variance tradeoff">Bias–variance tradeoff</a></li>

<li><a href="Computational_learning_theory" title="Computational learning theory">Computational learning theory</a></li>

<li><a href="Empirical_risk_minimization" title="Empirical risk minimization">Empirical risk minimization</a></li>

<li><a href="Occam_learning" title="Occam learning">Occam learning</a></li>

<li><a href="Probably_approximately_correct_learning" title="Probably approximately correct learning">PAC learning</a></li>

<li><a href="Statistical_learning_theory" title="Statistical learning theory">Statistical learning</a></li>

<li><a href="Vapnik%E2%80%93Chervonenkis_theory" title="Vapnik–Chervonenkis theory">VC theory</a></li>

<li><a href="Topological_deep_learning" title="Topological deep learning">Topological deep learning</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Journals and conferences</div><div class="sidebar-list-content mw-collapsible-content hlist">

<li><a href="Conference_on_Neural_Information_Processing_Systems" title="Conference on Neural Information Processing Systems">NeurIPS</a></li>

<li><a href="International_Joint_Conference_on_Artificial_Intelligence" title="International Joint Conference on Artificial Intelligence">IJCAI</a></li>

</tr><tr><td class="sidebar-content">

<div class="sidebar-list mw-collapsible mw-collapsed machine-learning-list-title"><div class="sidebar-list-title" style="border-top:1px solid #aaa; text-align:center;;color: var(--color-base)">Related articles</div><div class="sidebar-list-content mw-collapsible-content hlist">

<ul><li><a href="Glossary_of_artificial_intelligence" title="Glossary of artificial intelligence">Glossary of artificial intelligence</a></li>

<li><a href="List_of_datasets_for_machine-learning_research" title="List of datasets for machine-learning research">List of datasets for machine-learning research</a>

<ul><li><a href="List_of_datasets_in_computer_vision_and_image_processing" title="List of datasets in computer vision and image processing">List of datasets in computer vision and image processing</a></li></ul></li>

<li><a href="Outline_of_machine_learning" title="Outline of machine learning">Outline of machine learning</a></li></ul></div></div></td>

</tr><tr><td class="sidebar-navbar"><style data-mw-deduplicate="TemplateStyles:r1239400231">

/* start https://en.wikipedia.org/ */

.mw-parser-output .navbar{display:inline;font-size:88%;font-weight:normal}.mw-parser-output .navbar-collapse{float:left;text-align:left}.mw-parser-output .navbar-boxtext{word-spacing:0}.mw-parser-output .navbar ul{display:inline-block;white-space:nowrap;line-height:inherit}.mw-parser-output .navbar-brackets::before{margin-right:-0.125em;content:"[ "}.mw-parser-output .navbar-brackets::after{margin-left:-0.125em;content:" ]"}.mw-parser-output .navbar li{word-spacing:-0.125em}.mw-parser-output .navbar a>span,.mw-parser-output .navbar a>abbr{text-decoration:inherit}.mw-parser-output .navbar-mini abbr{font-variant:small-caps;border-bottom:none;text-decoration:none;cursor:inherit}.mw-parser-output .navbar-ct-full{font-size:114%;margin:0 7em}.mw-parser-output .navbar-ct-mini{font-size:114%;margin:0 4em}html.skin-theme-clientpref-night .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}@media(prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .navbar li a abbr{color:var(--color-base)!important}}@media print{.mw-parser-output .navbar{display:none!important}}

/* end https://en.wikipedia.org/ */

</style></td></tr></tbody></table>

In <a href="Statistics" title="Statistics">statistics</a>, an expectation–maximization (EM) algorithm is an <a href="Iterative_method" title="Iterative method">iterative method</a> to find (local) <a href="Maximum_likelihood" class="mw-redirect" title="Maximum likelihood">maximum likelihood</a> or <a href="Maximum_a_posteriori" class="mw-redirect" title="Maximum a posteriori">maximum a posteriori</a> (MAP) estimates of <a href="Parameter" title="Parameter">parameters</a> in <a href="Statistical_model" title="Statistical model">statistical models</a>, where the model depends on unobserved <a href="Latent_variable" class="mw-redirect" title="Latent variable">latent variables</a>.<a href="#cite_note-1">[1]</a> The EM iteration alternates between performing an expectation (E) step, which creates a function for the expectation of the <a href="Likelihood_function#Log-likelihood" title="Likelihood function">log-likelihood</a> evaluated using the current estimate for the parameters, and a maximization (M) step, which computes parameters maximizing the expected log-likelihood found on the E step. These parameter-estimates are then used to determine the distribution of the latent variables in the next E step. It can be used, for example, to estimate a mixture of <a href="Normal_distribution" title="Normal distribution">gaussians</a>, or to solve the multiple linear regression problem.<a href="#cite_note-2">[2]</a>

<div class="mw-heading mw-heading2"><h2 id="History">History</h2></div>

The EM algorithm was explained and given its name in a classic 1977 paper by <a href="Arthur_P._Dempster" title="Arthur P. Dempster">Arthur Dempster</a>, <a href="Nan_Laird" title="Nan Laird">Nan Laird</a>, and <a href="Donald_Rubin" title="Donald Rubin">Donald Rubin</a>.<a href="#cite_note-Dempster1977-3">[3]</a> They pointed out that the method had been "proposed many times in special circumstances" by earlier authors. One of the earliest is the gene-counting method for estimating allele frequencies by <a href="Cedric_Smith_(statistician)" title="Cedric Smith (statistician)">Cedric Smith</a>.<a href="#cite_note-4">[4]</a> Another was proposed by <a href="Herman_Otto_Hartley" title="Herman Otto Hartley">H.O. Hartley</a> in 1958, and Hartley and Hocking in 1977, from which many of the ideas in the Dempster–Laird–Rubin paper originated.<a href="#cite_note-5">[5]</a> Another one by S.K Ng, Thriyambakam Krishnan and G.J McLachlan in 1977.<a href="#cite_note-6">[6]</a> Hartley’s ideas can be broadened to any grouped discrete distribution. A very detailed treatment of the EM method for exponential families was published by Rolf Sundberg in his thesis and several papers,<a href="#cite_note-Sundberg1974-7">[7]</a><a href="#cite_note-Sundberg1971-8">[8]</a><a href="#cite_note-Sundberg1976-9">[9]</a> following his collaboration with <a href="Per_Martin-L%C3%B6f" title="Per Martin-Löf">Per Martin-Löf</a> and <a href="Anders_Martin-L%C3%B6f" title="Anders Martin-Löf">Anders Martin-Löf</a>.<a href="#cite_note-10">[10]</a><a href="#cite_note-Martin-Löf1966-11">[11]</a><a href="#cite_note-Martin-Löf1970-12">[12]</a><a href="#cite_note-Martin-Löf1974a-13">[13]</a><a href="#cite_note-Martin-Löf1974b-14">[14]</a> The Dempster–Laird–Rubin paper in 1977 generalized the method and sketched a convergence analysis for a wider class of problems. The Dempster–Laird–Rubin paper established the EM method as an important tool of statistical analysis. See also Meng and van Dyk (1997).

The convergence analysis of the Dempster–Laird–Rubin algorithm was flawed and a correct convergence analysis was published by <a href="C._F._Jeff_Wu" title="C. F. Jeff Wu">C. F. Jeff Wu</a> in 1983.<a href="#cite_note-Wu-15">[15]</a>

Wu's proof established the EM method's convergence also outside of the <a href="Exponential_family" title="Exponential family">exponential family</a>, as claimed by Dempster–Laird–Rubin.<a href="#cite_note-Wu-15">[15]</a>

<div class="mw-heading mw-heading2"><h2 id="Introduction">Introduction</h2></div>

The EM algorithm is used to find (local) <a href="Maximum_likelihood" class="mw-redirect" title="Maximum likelihood">maximum likelihood</a> parameters of a <a href="Statistical_model" title="Statistical model">statistical model</a> in cases where the equations cannot be solved directly. Typically these models involve <a href="Latent_variable" class="mw-redirect" title="Latent variable">latent variables</a> in addition to unknown <a href="Parameters" class="mw-redirect" title="Parameters">parameters</a> and known data observations. That is, either <a href="Missing_values" class="mw-redirect" title="Missing values">missing values</a> exist among the data, or the model can be formulated more simply by assuming the existence of further unobserved data points. For example, a <a href="Mixture_model" title="Mixture model">mixture model</a> can be described more simply by assuming that each observed data point has a corresponding unobserved data point, or latent variable, specifying the mixture component to which each data point belongs.

Finding a maximum likelihood solution typically requires taking the <a href="Derivative" title="Derivative">derivatives</a> of the <a href="Likelihood_function" title="Likelihood function">likelihood function</a> with respect to all the unknown values, the parameters and the latent variables, and simultaneously solving the resulting equations. In statistical models with latent variables, this is usually impossible. Instead, the result is typically a set of interlocking equations in which the solution to the parameters requires the values of the latent variables and vice versa, but substituting one set of equations into the other produces an unsolvable equation.

The EM algorithm proceeds from the observation that there is a way to solve these two sets of equations numerically. One can simply pick arbitrary values for one of the two sets of unknowns, use them to estimate the second set, then use these new values to find a better estimate of the first set, and then keep alternating between the two until the resulting values both converge to fixed points. It's not obvious that this will work, but it can be proven in this context. Additionally, it can be proven that the derivative of the likelihood is (arbitrarily close to) zero at that point, which in turn means that the point is either a local maximum or a <a href="Saddle_point" title="Saddle point">saddle point</a>.<a href="#cite_note-Wu-15">[15]</a> In general, multiple maxima may occur, with no guarantee that the global maximum will be found. Some likelihoods also have <a href="Mathematical_singularity" class="mw-redirect" title="Mathematical singularity">singularities</a> in them, i.e., nonsensical maxima. For example, one of the solutions that may be found by EM in a mixture model involves setting one of the components to have zero variance and the mean parameter for the same component to be equal to one of the data points.

<div class="mw-heading mw-heading2"><h2 id="Description">Description</h2></div>

<div class="mw-heading mw-heading3"><h3 id="The_symbols">The symbols</h3></div>

Given the <a href="Statistical_model" title="Statistical model">statistical model</a> which generates a set <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {X} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} }</annotation>

</semantics>

</math><img src="./9f75966a2f9d5672136fa9401ee1e75008f95ffd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.019ex; height:2.176ex;" alt="{\displaystyle \mathbf {X} }" loading="lazy"> of observed data, a set of unobserved latent data or <a href="Missing_values" class="mw-redirect" title="Missing values">missing values</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy">, and a vector of unknown parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy">, along with a <a href="Likelihood_function" title="Likelihood function">likelihood function</a> <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle L({\boldsymbol {\theta }};\mathbf {X} ,\mathbf {Z} )=p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})}">

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L({\boldsymbol {\theta }};\mathbf {X} ,\mathbf {Z} )=p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})}</annotation>

</semantics>

</math><img src="./d4a774eb5746942b5a8bb44632c83f9a50bf39ef.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:24.428ex; height:2.843ex;" alt="{\displaystyle L({\boldsymbol {\theta }};\mathbf {X} ,\mathbf {Z} )=p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})}" loading="lazy">, the <a href="Maximum_likelihood_estimate" class="mw-redirect" title="Maximum likelihood estimate">maximum likelihood estimate</a> (MLE) of the unknown parameters is determined by maximizing the <a href="Marginal_likelihood" title="Marginal likelihood">marginal likelihood</a> of the observed data

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L({\boldsymbol {\theta }};\mathbf {X} )=p(\mathbf {X} \mid {\boldsymbol {\theta }})=\int p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})\,d\mathbf {Z} =\int p(\mathbf {X} \mid \mathbf {Z} ,{\boldsymbol {\theta }})p(\mathbf {Z} \mid {\boldsymbol {\theta }})\,d\mathbf {Z} }</annotation>

</semantics>

</math><img src="./74ab2a5b6429f6d17632aedd043e6e177c9ca01c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.338ex; width:66.6ex; height:5.676ex;" alt="{\displaystyle L({\boldsymbol {\theta }};\mathbf {X} )=p(\mathbf {X} \mid {\boldsymbol {\theta }})=\int p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})\,d\mathbf {Z} =\int p(\mathbf {X} \mid \mathbf {Z} ,{\boldsymbol {\theta }})p(\mathbf {Z} \mid {\boldsymbol {\theta }})\,d\mathbf {Z} }" loading="lazy"></dd></dl>

However, this quantity is often intractable since <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> is unobserved and the distribution of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> is unknown before attaining <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy">.

<div class="mw-heading mw-heading3"><h3 id="The_EM_algorithm">The EM algorithm</h3></div>

The EM algorithm seeks to find the maximum likelihood estimate of the marginal likelihood by iteratively applying these two steps:

<dl><dd>Expectation step (E step): Define <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}</annotation>

</semantics>

</math><img src="./24d245ae1f551d986964bdcfa59ef4bad70079cb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.303ex; height:3.343ex;" alt="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}" loading="lazy"> as the <a href="Expected_value" title="Expected value">expected value</a> of the log <a href="Likelihood_function" title="Likelihood function">likelihood function</a> of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy">, with respect to the current <a href="Conditional_probability_distribution" title="Conditional probability distribution">conditional distribution</a> of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> given <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {X} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} }</annotation>

</semantics>

</math><img src="./9f75966a2f9d5672136fa9401ee1e75008f95ffd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.019ex; height:2.176ex;" alt="{\displaystyle \mathbf {X} }" loading="lazy"> and the current estimates of the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}^{(t)}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}^{(t)}}</annotation>

</semantics>

</math><img src="./4a1c52b206c71daeb0a1208f1f11182d5a90eb9b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.412ex; height:2.843ex;" alt="{\displaystyle {\boldsymbol {\theta }}^{(t)}}" loading="lazy">:

</mrow>

<msup>

</mrow>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})=\operatorname {E} _{\mathbf {Z} \sim p(\cdot |\mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}\left[\log p(\mathbf {X} ,\mathbf {Z} |{\boldsymbol {\theta }})\right]\,}</annotation>

</semantics>

</math><img src="./ceb93467728ae61d8611ec69c73ca0dc3a89dbd9.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.505ex; width:39.825ex; height:4.009ex;" alt="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})=\operatorname {E} _{\mathbf {Z} \sim p(\cdot |\mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}\left[\log p(\mathbf {X} ,\mathbf {Z} |{\boldsymbol {\theta }})\right]\,}" loading="lazy"></dd></dl></dd></dl>

<dl><dd>Maximization step (M step): Find the parameters that maximize this quantity:

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}^{(t+1)}={\underset {\boldsymbol {\theta }}{\operatorname {arg\,max} }}\ Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})\,}</annotation>

</semantics>

</math><img src="./ec0b048c1c3ec3993dda22954d5024c87c268c7d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -2.505ex; width:27.831ex; height:5.009ex;" alt="{\displaystyle {\boldsymbol {\theta }}^{(t+1)}={\underset {\boldsymbol {\theta }}{\operatorname {arg\,max} }}\ Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})\,}" loading="lazy"></dd></dl></dd></dl>

More succinctly, we can write it as one equation:<math display="block" xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}^{(t+1)}={\underset {\boldsymbol {\theta }}{\operatorname {arg\,max} }}\operatorname {E} _{\mathbf {Z} \sim p(\cdot |\mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}\left[\log p(\mathbf {X} ,\mathbf {Z} |{\boldsymbol {\theta }})\right]\,}">

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</munder>

</mrow>

<msub>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}^{(t+1)}={\underset {\boldsymbol {\theta }}{\operatorname {arg\,max} }}\operatorname {E} _{\mathbf {Z} \sim p(\cdot |\mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}\left[\log p(\mathbf {X} ,\mathbf {Z} |{\boldsymbol {\theta }})\right]\,}</annotation>

</semantics>

</math>

<div class="mw-heading mw-heading3"><h3 id="Interpretation_of_the_variables">Interpretation of the variables</h3></div>

The typical models to which EM is applied use <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> as a latent variable indicating membership in one of a set of groups:

<ol><li>The observed data points <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {X} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {X} }</annotation>

</semantics>

</math><img src="./9f75966a2f9d5672136fa9401ee1e75008f95ffd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:2.019ex; height:2.176ex;" alt="{\displaystyle \mathbf {X} }" loading="lazy"> may be <a href="Discrete_random_variable" class="mw-redirect" title="Discrete random variable">discrete</a> (taking values in a finite or countably infinite set) or <a href="Continuous_random_variable" class="mw-redirect" title="Continuous random variable">continuous</a> (taking values in an uncountably infinite set). Associated with each data point may be a vector of observations.</li>

<li>The <a href="Missing_values" class="mw-redirect" title="Missing values">missing values</a> (aka <a href="Latent_variables" class="mw-redirect" title="Latent variables">latent variables</a>) <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> are <a href="Discrete_random_variable" class="mw-redirect" title="Discrete random variable">discrete</a>, drawn from a fixed number of values, and with one latent variable per observed unit.</li>

<li>The parameters are continuous, and are of two kinds: Parameters that are associated with all data points, and those associated with a specific value of a latent variable (i.e., associated with all data points whose corresponding latent variable has that value).</li></ol>

However, it is possible to apply EM to other sorts of models.

The motivation is as follows. If the value of the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> is known, usually the value of the latent variables <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> can be found by maximizing the log-likelihood over all possible values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy">, either simply by iterating over <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> or through an algorithm such as the <a href="Viterbi_algorithm" title="Viterbi algorithm">Viterbi algorithm</a> for <a href="Hidden_Markov_model" title="Hidden Markov model">hidden Markov models</a>. Conversely, if we know the value of the latent variables <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy">, we can find an estimate of the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> fairly easily, typically by simply grouping the observed data points according to the value of the associated latent variable and averaging the values, or some function of the values, of the points in each group. This suggests an iterative algorithm, in the case where both <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> are unknown:

<ol><li>First, initialize the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> to some random values.</li>

<li>Compute the probability of each possible value of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> , given <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy">.</li>

<li>Then, use the just-computed values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> to compute a better estimate for the parameters <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy">.</li>

<li>Iterate steps 2 and 3 until convergence.</li></ol>

The algorithm as just described monotonically approaches a local minimum of the cost function.

<div class="mw-heading mw-heading2"><h2 id="Properties">Properties</h2></div>

Although an EM iteration does increase the observed data (i.e., marginal) likelihood function, no guarantee exists that the sequence converges to a <a href="Maximum_likelihood_estimator" class="mw-redirect" title="Maximum likelihood estimator">maximum likelihood estimator</a>. For <a href="Bimodal_distribution" class="mw-redirect" title="Bimodal distribution">multimodal distributions</a>, this means that an EM algorithm may converge to a <a href="Local_maximum" class="mw-redirect" title="Local maximum">local maximum</a> of the observed data likelihood function, depending on starting values. A variety of heuristic or <a href="Metaheuristic" title="Metaheuristic">metaheuristic</a> approaches exist to escape a local maximum, such as random-restart <a href="Hill_climbing" title="Hill climbing">hill climbing</a> (starting with several different random initial estimates <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}^{(t)}}">

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}^{(t)}}</annotation>

</semantics>

</math><img src="./4a1c52b206c71daeb0a1208f1f11182d5a90eb9b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.412ex; height:2.843ex;" alt="{\displaystyle {\boldsymbol {\theta }}^{(t)}}" loading="lazy">), or applying <a href="Simulated_annealing" title="Simulated annealing">simulated annealing</a> methods.

EM is especially useful when the likelihood is an <a href="Exponential_family" title="Exponential family">exponential family</a>, see Sundberg (2019, Ch. 8) for a comprehensive treatment:<a href="#cite_note-16">[16]</a> the E step becomes the sum of expectations of <a href="Sufficient_statistic" title="Sufficient statistic">sufficient statistics</a>, and the M step involves maximizing a linear function. In such a case, it is usually possible to derive <a href="Closed-form_expression" title="Closed-form expression">closed-form expression</a> updates for each step, using the Sundberg formula<a href="#cite_note-17">[17]</a> (proved and published by Rolf Sundberg, based on unpublished results of <a href="Per_Martin-L%C3%B6f" title="Per Martin-Löf">Per Martin-Löf</a> and <a href="Anders_Martin-L%C3%B6f" title="Anders Martin-Löf">Anders Martin-Löf</a>).<a href="#cite_note-Sundberg1971-8">[8]</a><a href="#cite_note-Sundberg1976-9">[9]</a><a href="#cite_note-Martin-Löf1966-11">[11]</a><a href="#cite_note-Martin-Löf1970-12">[12]</a><a href="#cite_note-Martin-Löf1974a-13">[13]</a><a href="#cite_note-Martin-Löf1974b-14">[14]</a>

The EM method was modified to compute <a href="Maximum_a_posteriori" class="mw-redirect" title="Maximum a posteriori">maximum a posteriori</a> (MAP) estimates for <a href="Bayesian_inference" title="Bayesian inference">Bayesian inference</a> in the original paper by Dempster, Laird, and Rubin.

Other methods exist to find maximum likelihood estimates, such as <a href="Gradient_descent" title="Gradient descent">gradient descent</a>, <a href="Conjugate_gradient" class="mw-redirect" title="Conjugate gradient">conjugate gradient</a>, or variants of the <a href="Gauss%E2%80%93Newton_algorithm" title="Gauss–Newton algorithm">Gauss–Newton algorithm</a>. Unlike EM, such methods typically require the evaluation of first and/or second derivatives of the likelihood function.

<div class="mw-heading mw-heading2"><h2 id="Proof_of_correctness">Proof of correctness</h2></div>

Expectation-Maximization works to improve <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}</annotation>

</semantics>

</math><img src="./24d245ae1f551d986964bdcfa59ef4bad70079cb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.303ex; height:3.343ex;" alt="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}" loading="lazy"> rather than directly improving <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})}</annotation>

</semantics>

</math><img src="./3b1c4eed3feeeca1f0a16cf8a0dd57b2ecc5e6e1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:11.601ex; height:2.843ex;" alt="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})}" loading="lazy">. Here it is shown that improvements to the former imply improvements to the latter.<a href="#cite_note-Little1987-18">[18]</a>

For any <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> with non-zero probability <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }})}">

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }})}</annotation>

</semantics>

</math><img src="./4921aead43c89a940741fbce1a43b20ee9f40aa2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:10.999ex; height:2.843ex;" alt="{\displaystyle p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }})}" loading="lazy">, we can write

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})=\log p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})-\log p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}).}</annotation>

</semantics>

</math><img src="./8799d6e07102db446f9348b163903df2eb51c0f7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:46.724ex; height:2.843ex;" alt="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})=\log p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})-\log p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}).}" loading="lazy"></dd></dl>

We take the expectation over possible values of the unknown data <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy"> under the current parameter estimate <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \theta ^{(t)}}">

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta ^{(t)}}</annotation>

</semantics>

</math><img src="./7dbe79968a27c7b3508260e53b0123a5872b16cc.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:3.196ex; height:2.843ex;" alt="{\displaystyle \theta ^{(t)}}" loading="lazy"> by multiplying both sides by <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}">

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}</annotation>

</semantics>

</math><img src="./a7c37375a0cfaacb347c7d16a264ff7556aa4607.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; margin-left: -0.089ex; width:13.104ex; height:3.343ex;" alt="{\displaystyle p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})}" loading="lazy"> and summing (or integrating) over <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {Z} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {Z} }</annotation>

</semantics>

</math><img src="./b776aaf12c2da4b78ca777cb8295c2000bfd51f5.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.634ex; height:2.176ex;" alt="{\displaystyle \mathbf {Z} }" loading="lazy">. The left-hand side is the expectation of a constant, so we get:

<mtr>

<mtd>

</mrow>

</mrow>

</mtd>

<mtd>

</mrow>

</munder>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</mrow>

</munder>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}\log p(\mathbf {X} \mid {\boldsymbol {\theta }})&=\sum _{\mathbf {Z} }p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})\log p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})-\sum _{\mathbf {Z} }p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})\log p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }})\\&=Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})+H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)}),\end{aligned}}}</annotation>

</semantics>

</math><img src="./dea8d338c63fb03728cbc5188d3f7fe59d89fcee.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -4.005ex; width:81.117ex; height:9.176ex;" alt="{\displaystyle {\begin{aligned}\log p(\mathbf {X} \mid {\boldsymbol {\theta }})&=\sum _{\mathbf {Z} }p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})\log p(\mathbf {X} ,\mathbf {Z} \mid {\boldsymbol {\theta }})-\sum _{\mathbf {Z} }p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }}^{(t)})\log p(\mathbf {Z} \mid \mathbf {X} ,{\boldsymbol {\theta }})\\&=Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})+H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)}),\end{aligned}}}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}</annotation>

</semantics>

</math><img src="./002e61b049741e19c9fb5b0c331b1ec093e334bd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.528ex; height:3.343ex;" alt="{\displaystyle H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}" loading="lazy"> is defined by the negated sum it is replacing.

This last equation holds for every value of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> including <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}={\boldsymbol {\theta }}^{(t)}}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}={\boldsymbol {\theta }}^{(t)}}</annotation>

</semantics>

</math><img src="./e25b6d9fac398c249453f943de46eb5b7cac8a09.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:7.817ex; height:2.843ex;" alt="{\displaystyle {\boldsymbol {\theta }}={\boldsymbol {\theta }}^{(t)}}" loading="lazy">,

</mrow>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }}^{(t)})=Q({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})+H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)}),}</annotation>

</semantics>

</math><img src="./49af4819ab3c959837e17ed91605b592e863cf95.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:45.334ex; height:3.343ex;" alt="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }}^{(t)})=Q({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})+H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)}),}" loading="lazy"></dd></dl>

and subtracting this last equation from the previous equation gives

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})-\log p(\mathbf {X} \mid {\boldsymbol {\theta }}^{(t)})=Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})-Q({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})+H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})-H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)}).}</annotation>

</semantics>

</math><img src="./c8ad176b3f297b826026386893cbea4585678dfe.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:86.287ex; height:3.343ex;" alt="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})-\log p(\mathbf {X} \mid {\boldsymbol {\theta }}^{(t)})=Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})-Q({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})+H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})-H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)}).}" loading="lazy"></dd></dl>

However, <a href="Gibbs'_inequality" title="Gibbs' inequality">Gibbs' inequality</a> tells us that <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})\geq H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})\geq H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})}</annotation>

</semantics>

</math><img src="./507c40004ea5b6171086fd4b4c0cd25d82b8cf5c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:26.26ex; height:3.343ex;" alt="{\displaystyle H({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})\geq H({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)})}" loading="lazy">, so we can conclude that

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})-\log p(\mathbf {X} \mid {\boldsymbol {\theta }}^{(t)})\geq Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})-Q({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)}).}</annotation>

</semantics>

</math><img src="./8eb0a8b9d3317d3f24df360b699fd6da686ccbf4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:57.444ex; height:3.343ex;" alt="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})-\log p(\mathbf {X} \mid {\boldsymbol {\theta }}^{(t)})\geq Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})-Q({\boldsymbol {\theta }}^{(t)}\mid {\boldsymbol {\theta }}^{(t)}).}" loading="lazy"></dd></dl>

In words, choosing <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\boldsymbol {\theta }}}">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\theta }}}</annotation>

</semantics>

</math><img src="./33b025a6bf54ec02e65c871dc3e5897c921419cf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.306ex; height:2.176ex;" alt="{\displaystyle {\boldsymbol {\theta }}}" loading="lazy"> to improve <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}">

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}</annotation>

</semantics>

</math><img src="./24d245ae1f551d986964bdcfa59ef4bad70079cb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:10.303ex; height:3.343ex;" alt="{\displaystyle Q({\boldsymbol {\theta }}\mid {\boldsymbol {\theta }}^{(t)})}" loading="lazy"> causes <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})}">

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})}</annotation>

</semantics>

</math><img src="./3b1c4eed3feeeca1f0a16cf8a0dd57b2ecc5e6e1.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:11.601ex; height:2.843ex;" alt="{\displaystyle \log p(\mathbf {X} \mid {\boldsymbol {\theta }})}" loading="lazy"> to improve at least as much.

<div class="mw-heading mw-heading2"><h2 id="As_a_maximization–maximization_procedure">As a maximization–maximization procedure</h2></div>

The EM algorithm can be viewed as two alternating maximization steps, that is, as an example of <a href="Coordinate_descent" title="Coordinate descent">coordinate descent</a>.<a href="#cite_note-neal1999-19">[19]</a><a href="#cite_note-hastie2001-20">[20]</a> Consider the function:

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F(q,\theta ):=\operatorname {E} _{q}[\log L(\theta ;x,Z)]+H(q),}</annotation>

</semantics>

</math><img src="./f4e0798985f246154015deb375e1494dfa734922.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:35.704ex; height:3.009ex;" alt="{\displaystyle F(q,\theta ):=\operatorname {E} _{q}[\log L(\theta ;x,Z)]+H(q),}" loading="lazy"></dd></dl>

where q is an arbitrary probability distribution over the unobserved data z and H(q) is the <a href="Entropy_(information_theory)" title="Entropy (information theory)">entropy</a> of the distribution q. This function can be written as

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F(q,\theta )=-D_{\mathrm {KL} }{\big (}q\parallel p_{Z\mid X}(\cdot \mid x;\theta ){\big )}+\log L(\theta ;x),}</annotation>

</semantics>

</math><img src="./57e86296710dd8d5759ee0d751bb662620b2c13c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:47.752ex; height:3.343ex;" alt="{\displaystyle F(q,\theta )=-D_{\mathrm {KL} }{\big (}q\parallel p_{Z\mid X}(\cdot \mid x;\theta ){\big )}+\log L(\theta ;x),}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle p_{Z\mid X}(\cdot \mid x;\theta )}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle p_{Z\mid X}(\cdot \mid x;\theta )}</annotation>

</semantics>

</math><img src="./5e674aed2f32feeb6eb547964af3618c9c2a9b24.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; margin-left: -0.089ex; width:12.384ex; height:3.176ex;" alt="{\displaystyle p_{Z\mid X}(\cdot \mid x;\theta )}" loading="lazy"> is the conditional distribution of the unobserved data given the observed data <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle x}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle x}</annotation>

</semantics>

</math><img src="./87f9e315fd7e2ba406057a97300593c4802b53e4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.33ex; height:1.676ex;" alt="{\displaystyle x}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle D_{KL}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle D_{KL}}</annotation>

</semantics>

</math><img src="./b6eb7f08e9495f990043ebb7a3da95cb9bdb37f2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:4.737ex; height:2.509ex;" alt="{\displaystyle D_{KL}}" loading="lazy"> is the <a href="Kullback%E2%80%93Leibler_divergence" title="Kullback–Leibler divergence">Kullback–Leibler divergence</a>.

Then the steps in the EM algorithm may be viewed as:

<dl><dd>Expectation step: Choose <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle q}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q}</annotation>

</semantics>

</math><img src="./06809d64fa7c817ffc7e323f85997f783dbdf71d.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.07ex; height:2.009ex;" alt="{\displaystyle q}" loading="lazy"> to maximize <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle F}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F}</annotation>

</semantics>

</math><img src="./545fd099af8541605f7ee55f08225526be88ce57.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.741ex; height:2.176ex;" alt="{\displaystyle F}" loading="lazy">:

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle q^{(t)}=\operatorname {arg\,max} _{q}\ F(q,\theta ^{(t)})}</annotation>

</semantics>

</math><img src="./ec388559464783fb600754a399c9a6172fa10bee.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.171ex; width:25.038ex; height:3.676ex;" alt="{\displaystyle q^{(t)}=\operatorname {arg\,max} _{q}\ F(q,\theta ^{(t)})}" loading="lazy"></dd></dl></dd>

<dd>Maximization step: Choose <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \theta }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta }</annotation>

</semantics>

</math><img src="./6e5ab2664b422d53eb0c7df3b87e1360d75ad9af.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.09ex; height:2.176ex;" alt="{\displaystyle \theta }" loading="lazy"> to maximize <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle F}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle F}</annotation>

</semantics>

</math><img src="./545fd099af8541605f7ee55f08225526be88ce57.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.741ex; height:2.176ex;" alt="{\displaystyle F}" loading="lazy">:

<msup>

</mrow>

</msup>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta ^{(t+1)}=\operatorname {arg\,max} _{\theta }\ F(q^{(t)},\theta )}</annotation>

</semantics>

</math><img src="./696d87c8260b868730707884aa48c0db4bbaaae4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:27.175ex; height:3.343ex;" alt="{\displaystyle \theta ^{(t+1)}=\operatorname {arg\,max} _{\theta }\ F(q^{(t)},\theta )}" loading="lazy"></dd></dl></dd></dl>

<div class="mw-heading mw-heading2"><h2 id="Applications">Applications</h2></div>

<ul><li>EM is frequently used for <a href="Parameter_estimation" class="mw-redirect" title="Parameter estimation">parameter estimation</a> of <a href="Mixed_model" title="Mixed model">mixed models</a>,<a href="#cite_note-21">[21]</a><a href="#cite_note-22">[22]</a> notably in <a href="Quantitative_genetics" title="Quantitative genetics">quantitative genetics</a>.<a href="#cite_note-23">[23]</a></li>

<li>In <a href="Psychometrics" title="Psychometrics">psychometrics</a>, EM is an important tool for estimating item parameters and latent abilities of <a href="Item_response_theory" title="Item response theory">item response theory</a> models.</li>

<li>With the ability to deal with missing data and observe unidentified variables, EM is becoming a useful tool to price and manage risk of a portfolio.</li>

<li>The EM algorithm (and its faster variant <a href="Ordered_subset_expectation_maximization" title="Ordered subset expectation maximization">ordered subset expectation maximization</a>) is also widely used in <a href="Medical_imaging" title="Medical imaging">medical image</a> reconstruction, especially in <a href="Positron_emission_tomography" title="Positron emission tomography">positron emission tomography</a>, <a href="Single-photon_emission_computed_tomography" title="Single-photon emission computed tomography">single-photon emission computed tomography</a>, and x-ray <a href="Computed_tomography" class="mw-redirect" title="Computed tomography">computed tomography</a>. See below for other faster variants of EM.</li>

<li>In <a href="Structural_engineering" title="Structural engineering">structural engineering</a>, the Structural Identification using Expectation Maximization (STRIDE)<a href="#cite_note-24">[24]</a> algorithm is an output-only method for identifying natural vibration properties of a structural system using sensor data (see <a href="Operational_Modal_Analysis" class="mw-redirect" title="Operational Modal Analysis">Operational Modal Analysis</a>).</li>

<li>EM is also used for <a href="Data_clustering" class="mw-redirect" title="Data clustering">data clustering</a>. In <a href="Natural_language_processing" title="Natural language processing">natural language processing</a>, two prominent instances of the algorithm are the <a href="Baum%E2%80%93Welch_algorithm" title="Baum–Welch algorithm">Baum–Welch algorithm</a> for <a href="Hidden_Markov_models" class="mw-redirect" title="Hidden Markov models">hidden Markov models</a>, and the <a href="Inside-outside_algorithm" class="mw-redirect" title="Inside-outside algorithm">inside-outside algorithm</a> for unsupervised induction of <a href="Probabilistic_context-free_grammar" title="Probabilistic context-free grammar">probabilistic context-free grammars</a>.</li>

<li>In the analysis of intertrade <a href="Waiting_time" class="mw-redirect" title="Waiting time">waiting times</a> i.e. the time between subsequent trades in <a href="Share_(finance)" title="Share (finance)">shares of stock</a> at a <a href="Stock_exchange" title="Stock exchange">stock exchange</a> the EM algorithm has proved to be very useful.<a href="#cite_note-25">[25]</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="Filtering_and_smoothing_EM_algorithms">Filtering and smoothing EM algorithms</h2></div>

A <a href="Kalman_filter" title="Kalman filter">Kalman filter</a> is typically used for on-line state estimation and a minimum-variance smoother may be employed for off-line or batch state estimation. However, these minimum-variance solutions require estimates of the state-space model parameters. EM algorithms can be used for solving joint state and parameter estimation problems.

Filtering and smoothing EM algorithms arise by repeating this two-step procedure:

<dd>Operate a Kalman filter or a minimum-variance smoother designed with current parameter estimates to obtain updated state estimates.</dd></dl>

<dd>Use the filtered or smoothed state estimates within maximum-likelihood calculations to obtain updated parameter estimates.</dd></dl>

Suppose that a <a href="Kalman_filter" title="Kalman filter">Kalman filter</a> or minimum-variance smoother operates on measurements of a single-input-single-output system that possess additive white noise. An updated measurement noise variance estimate can be obtained from the <a href="Maximum_likelihood" class="mw-redirect" title="Maximum likelihood">maximum likelihood</a> calculation

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msubsup>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msup>

<msub>

</mrow>

</msub>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widehat {\sigma }}_{v}^{2}={\frac {1}{N}}\sum _{k=1}^{N}{(z_{k}-{\widehat {x}}_{k})}^{2},}</annotation>

</semantics>

</math><img src="./1beadd771d6bc57b9ea869b4310ebbff6bc4eba2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:23.496ex; height:7.343ex;" alt="{\displaystyle {\widehat {\sigma }}_{v}^{2}={\frac {1}{N}}\sum _{k=1}^{N}{(z_{k}-{\widehat {x}}_{k})}^{2},}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\widehat {x}}_{k}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widehat {x}}_{k}}</annotation>

</semantics>

</math><img src="./a18153ee206f0216802fb91fcaed7ba3a170490a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.464ex; height:2.676ex;" alt="{\displaystyle {\widehat {x}}_{k}}" loading="lazy"> are scalar output estimates calculated by a filter or a smoother from N scalar measurements <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle z_{k}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle z_{k}}</annotation>

</semantics>

</math><img src="./a51cdfac24f8b95ed711f11ea9502da4087b6a24.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.17ex; height:2.009ex;" alt="{\displaystyle z_{k}}" loading="lazy">. The above update can also be applied to updating a Poisson measurement noise intensity. Similarly, for a first-order auto-regressive process, an updated process noise variance estimate can be calculated by

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msubsup>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

<msup>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<mover>

</mover>

</mrow>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widehat {\sigma }}_{w}^{2}={\frac {1}{N}}\sum _{k=1}^{N}{({\widehat {x}}_{k+1}-{\widehat {F}}{\widehat {x}}_{k})}^{2},}</annotation>

</semantics>

</math><img src="./d4cd67d275b9d98918febdc2bd14f4085d12dcaf.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.005ex; width:28.116ex; height:7.343ex;" alt="{\displaystyle {\widehat {\sigma }}_{w}^{2}={\frac {1}{N}}\sum _{k=1}^{N}{({\widehat {x}}_{k+1}-{\widehat {F}}{\widehat {x}}_{k})}^{2},}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\widehat {x}}_{k}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widehat {x}}_{k}}</annotation>

</semantics>

</math><img src="./a18153ee206f0216802fb91fcaed7ba3a170490a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.464ex; height:2.676ex;" alt="{\displaystyle {\widehat {x}}_{k}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle {\widehat {x}}_{k+1}}">

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widehat {x}}_{k+1}}</annotation>

</semantics>

</math><img src="./74df0419c9808d257d9a4addec6ee67bb8b3d2f3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:4.565ex; height:2.676ex;" alt="{\displaystyle {\widehat {x}}_{k+1}}" loading="lazy"> are scalar state estimates calculated by a filter or a smoother. The updated model coefficient estimate is obtained via

<mover>

</mover>

</mrow>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

<msup>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

<mover>

</mover>

</mrow>

<msub>

<mover>

</mover>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

<mover>

</mover>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\widehat {F}}={\frac {\sum _{k=1}^{N}{({\widehat {x}}_{k+1}-{\widehat {F}}{\widehat {x}}_{k})}^{2}}{\sum _{k=1}^{N}{\widehat {x}}_{k}^{2}}}.}</annotation>

</semantics>

</math><img src="./6136b1f02bee9d8cb762fc508fc951bfc46f49ef.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.171ex; width:27.085ex; height:8.009ex;" alt="{\displaystyle {\widehat {F}}={\frac {\sum _{k=1}^{N}{({\widehat {x}}_{k+1}-{\widehat {F}}{\widehat {x}}_{k})}^{2}}{\sum _{k=1}^{N}{\widehat {x}}_{k}^{2}}}.}" loading="lazy"></dd></dl>

The convergence of parameter estimates such as those above are well studied.<a href="#cite_note-26">[26]</a><a href="#cite_note-27">[27]</a><a href="#cite_note-28">[28]</a><a href="#cite_note-29">[29]</a>

<div class="mw-heading mw-heading2"><h2 id="Variants">Variants</h2></div>

A number of methods have been proposed to accelerate the sometimes slow convergence of the EM algorithm, such as those using <a href="Conjugate_gradient" class="mw-redirect" title="Conjugate gradient">conjugate gradient</a> and modified <a href="Newton's_method" title="Newton's method">Newton's methods</a> (Newton–Raphson).<a href="#cite_note-30">[30]</a> Also, EM can be used with constrained estimation methods.

Parameter-expanded expectation maximization (PX-EM) algorithm often provides speed up by "us[ing] a `covariance adjustment' to correct the analysis of the M step, capitalising on extra information captured in the imputed complete data".<a href="#cite_note-31">[31]</a>

Expectation conditional maximization (ECM) replaces each M step with a sequence of conditional maximization (CM) steps in which each parameter θi is maximized individually, conditionally on the other parameters remaining fixed.<a href="#cite_note-32">[32]</a> Itself can be extended into the Expectation conditional maximization either (ECME) algorithm.<a href="#cite_note-33">[33]</a>

This idea is further extended in generalized expectation maximization (GEM) algorithm, in which is sought only an increase in the objective function F for both the E step and M step as described in the <a href="#As_a_maximization–maximization_procedure">As a maximization–maximization procedure</a> section.<a href="#cite_note-neal1999-19">[19]</a> GEM is further developed in a distributed environment and shows promising results.<a href="#cite_note-34">[34]</a>

It is also possible to consider the EM algorithm as a subclass of the <a href="MM_algorithm" title="MM algorithm">MM</a> (Majorize/Minimize or Minorize/Maximize, depending on context) algorithm,<a href="#cite_note-35">[35]</a> and therefore use any machinery developed in the more general case.

<div class="mw-heading mw-heading3"><h3 id="α-EM_algorithm">α-EM algorithm</h3></div>

The Q-function used in the EM algorithm is based on the log likelihood. Therefore, it is regarded as the log-EM algorithm. The use of the log likelihood can be generalized to that of the α-log likelihood ratio. Then, the α-log likelihood ratio of the observed data can be exactly expressed as equality by using the Q-function of the α-log likelihood ratio and the α-divergence. Obtaining this Q-function is a generalized E step. Its maximization is a generalized M step. This pair is called the α-EM algorithm<a href="#cite_note-36">[36]</a>

which contains the log-EM algorithm as its subclass. Thus, the α-EM algorithm by <a href="Yasuo_Matsuyama" title="Yasuo Matsuyama">Yasuo Matsuyama</a> is an exact generalization of the log-EM algorithm. No computation of gradient or Hessian matrix is needed. The α-EM shows faster convergence than the log-EM algorithm by choosing an appropriate α. The α-EM algorithm leads to a faster version of the Hidden Markov model estimation algorithm α-HMM.

<div class="mw-heading mw-heading2"><h2 id="Relation_to_variational_Bayes_methods">Relation to variational Bayes methods</h2></div>

EM is a partially non-Bayesian, maximum likelihood method. Its final result gives a <a href="Probability_distribution" title="Probability distribution">probability distribution</a> over the latent variables (in the Bayesian style) together with a point estimate for θ (either a <a href="Maximum_likelihood_estimation" title="Maximum likelihood estimation">maximum likelihood estimate</a> or a posterior mode). A fully Bayesian version of this may be wanted, giving a probability distribution over θ and the latent variables. The Bayesian approach to inference is simply to treat θ as another latent variable. In this paradigm, the distinction between the E and M steps disappears. If using the factorized Q approximation as described above (<a href="Variational_Bayes" class="mw-redirect" title="Variational Bayes">variational Bayes</a>), solving can iterate over each latent variable (now including θ) and optimize them one at a time. Now, k steps per iteration are needed, where k is the number of latent variables. For <a href="Graphical_models" class="mw-redirect" title="Graphical models">graphical models</a> this is easy to do as each variable's new Q depends only on its <a href="Markov_blanket" title="Markov blanket">Markov blanket</a>, so local <a href="Message_passing_(disambiguation)" class="mw-disambig" title="Message passing (disambiguation)">message passing</a> can be used for efficient inference.

<div class="mw-heading mw-heading2"><h2 id="Geometric_interpretation">Geometric interpretation</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}@media print{body.ns-0 .mw-parser-output .hatnote{display:none!important}}

/* end https://en.wikipedia.org/ */

</style><div role="note" class="hatnote navigation-not-searchable">Further information: <a href="Information_geometry" title="Information geometry">Information geometry</a></div>

In <a href="Information_geometry" title="Information geometry">information geometry</a>, the E step and the M step are interpreted as projections under dual <a href="Affine_connection" title="Affine connection">affine connections</a>, called the e-connection and the m-connection; the <a href="Kullback%E2%80%93Leibler_divergence" title="Kullback–Leibler divergence">Kullback–Leibler divergence</a> can also be understood in these terms.

<div class="mw-heading mw-heading2"><h2 id="Examples">Examples</h2></div>

<div class="mw-heading mw-heading3"><h3 id="Gaussian_mixture">Gaussian mixture</h3></div>

Let <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {x} =(\mathbf {x} _{1},\mathbf {x} _{2},\ldots ,\mathbf {x} _{n})}">

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {x} =(\mathbf {x} _{1},\mathbf {x} _{2},\ldots ,\mathbf {x} _{n})}</annotation>

</semantics>

</math><img src="./61e9345a73e716032dfe920da2ef9d9265e6226b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:20.091ex; height:2.843ex;" alt="{\displaystyle \mathbf {x} =(\mathbf {x} _{1},\mathbf {x} _{2},\ldots ,\mathbf {x} _{n})}" loading="lazy"> be a sample of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle n}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle n}</annotation>

</semantics>

</math><img src="./a601995d55609f2d9f5e233e36fbe9ea26011b3b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.395ex; height:1.676ex;" alt="{\displaystyle n}" loading="lazy"> independent observations from a <a href="Mixture_model" title="Mixture model">mixture</a> of two <a href="Multivariate_normal_distribution" title="Multivariate normal distribution">multivariate normal distributions</a> of dimension <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle d}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle d}</annotation>

</semantics>

</math><img src="./e85ff03cbe0c7341af6b982e47e9f90d235c66ab.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.216ex; height:2.176ex;" alt="{\displaystyle d}" loading="lazy">, and let <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {z} =(z_{1},z_{2},\ldots ,z_{n})}">

</mrow>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {z} =(z_{1},z_{2},\ldots ,z_{n})}</annotation>

</semantics>

</math><img src="./394a19c373675e608117295cd6415e84f06eaaeb.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:18.878ex; height:2.843ex;" alt="{\displaystyle \mathbf {z} =(z_{1},z_{2},\ldots ,z_{n})}" loading="lazy"> be the latent variables that determine the component from which the observation originates.<a href="#cite_note-hastie2001-20">[20]</a>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}\mid (Z_{i}=1)\sim {\mathcal {N}}_{d}({\boldsymbol {\mu }}_{1},\Sigma _{1})}</annotation>

</semantics>

</math><img src="./e88d94caf64a6cc0cd17d33538f6dbf959fa023b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:27.49ex; height:3.009ex;" alt="{\displaystyle X_{i}\mid (Z_{i}=1)\sim {\mathcal {N}}_{d}({\boldsymbol {\mu }}_{1},\Sigma _{1})}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle X_{i}\mid (Z_{i}=2)\sim {\mathcal {N}}_{d}({\boldsymbol {\mu }}_{2},\Sigma _{2}),}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle X_{i}\mid (Z_{i}=2)\sim {\mathcal {N}}_{d}({\boldsymbol {\mu }}_{2},\Sigma _{2}),}</annotation>

</semantics>

</math><img src="./f9513f8d71de99a9b05902f3cffd3f3b731c7c93.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:28.137ex; height:3.009ex;" alt="{\displaystyle X_{i}\mid (Z_{i}=2)\sim {\mathcal {N}}_{d}({\boldsymbol {\mu }}_{2},\Sigma _{2}),}" loading="lazy"></dd></dl>

where

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {P} (Z_{i}=1)=\tau _{1}\,}</annotation>

</semantics>

</math><img src="./9ee708039dc8c18644060681372da6736ecec2ba.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:15.596ex; height:2.843ex;" alt="{\displaystyle \operatorname {P} (Z_{i}=1)=\tau _{1}\,}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \operatorname {P} (Z_{i}=2)=\tau _{2}=1-\tau _{1}.}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \operatorname {P} (Z_{i}=2)=\tau _{2}=1-\tau _{1}.}</annotation>

</semantics>

</math><img src="./69d5d20010a5d405950bcddfeee65fe0e4720e85.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:25.028ex; height:2.843ex;" alt="{\displaystyle \operatorname {P} (Z_{i}=2)=\tau _{2}=1-\tau _{1}.}" loading="lazy"></dd></dl>

The aim is to estimate the unknown parameters representing the mixing value between the Gaussians and the means and covariances of each:

</mrow>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta ={\big (}{\boldsymbol {\tau }},{\boldsymbol {\mu }}_{1},{\boldsymbol {\mu }}_{2},\Sigma _{1},\Sigma _{2}{\big )},}</annotation>

</semantics>

</math><img src="./bc9e5920c91b1e981bfd29ff579a9015aed036e2.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:23.383ex; height:3.176ex;" alt="{\displaystyle \theta ={\big (}{\boldsymbol {\tau }},{\boldsymbol {\mu }}_{1},{\boldsymbol {\mu }}_{2},\Sigma _{1},\Sigma _{2}{\big )},}" loading="lazy"></dd></dl>

where the incomplete-data likelihood function is

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L(\theta ;\mathbf {x} )=\prod _{i=1}^{n}\sum _{j=1}^{2}\tau _{j}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{j},\Sigma _{j}),}</annotation>

</semantics>

</math><img src="./d298ae52eb7b86ed22202288fbf66da466994d60.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.338ex; width:32.787ex; height:7.676ex;" alt="{\displaystyle L(\theta ;\mathbf {x} )=\prod _{i=1}^{n}\sum _{j=1}^{2}\tau _{j}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{j},\Sigma _{j}),}" loading="lazy"></dd></dl>

and the complete-data likelihood function is

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L(\theta ;\mathbf {x} ,\mathbf {z} )=p(\mathbf {x} ,\mathbf {z} \mid \theta )=\prod _{i=1}^{n}\prod _{j=1}^{2}\ [f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{j},\Sigma _{j})\tau _{j}]^{\mathbb {I} (z_{i}=j)},}</annotation>

</semantics>

</math><img src="./8992a646a56612c1c52a38302e6a84ccceeb949a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.338ex; width:54.152ex; height:7.676ex;" alt="{\displaystyle L(\theta ;\mathbf {x} ,\mathbf {z} )=p(\mathbf {x} ,\mathbf {z} \mid \theta )=\prod _{i=1}^{n}\prod _{j=1}^{2}\ [f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{j},\Sigma _{j})\tau _{j}]^{\mathbb {I} (z_{i}=j)},}" loading="lazy"></dd></dl>

or

</mrow>

</mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mstyle>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

<mfrac>

</mfrac>

</mstyle>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<mfrac>

</mfrac>

</mstyle>

</mrow>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle L(\theta ;\mathbf {x} ,\mathbf {z} )=\exp \left\{\sum _{i=1}^{n}\sum _{j=1}^{2}\mathbb {I} (z_{i}=j){\big [}\log \tau _{j}-{\tfrac {1}{2}}\log |\Sigma _{j}|-{\tfrac {1}{2}}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})^{\top }\Sigma _{j}^{-1}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})-{\tfrac {d}{2}}\log(2\pi ){\big ]}\right\},}</annotation>

</semantics>

</math><img src="./c7c30db649a3d79fca184a9ba9485609ff6ddf62.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.338ex; width:97.116ex; height:7.676ex;" alt="{\displaystyle L(\theta ;\mathbf {x} ,\mathbf {z} )=\exp \left\{\sum _{i=1}^{n}\sum _{j=1}^{2}\mathbb {I} (z_{i}=j){\big [}\log \tau _{j}-{\tfrac {1}{2}}\log |\Sigma _{j}|-{\tfrac {1}{2}}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})^{\top }\Sigma _{j}^{-1}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})-{\tfrac {d}{2}}\log(2\pi ){\big ]}\right\},}" loading="lazy"></dd></dl>

where <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {I} }">

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {I} }</annotation>

</semantics>

</math><img src="./8205f06e0d279689ed04a1ac04a3d9c249c637df.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:0.905ex; height:2.176ex;" alt="{\displaystyle \mathbb {I} }" loading="lazy"> is an <a href="Indicator_function" title="Indicator function">indicator function</a> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle f}">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle f}</annotation>

</semantics>

</math><img src="./132e57acb643253e7810ee9702d9581f159a1c61.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:1.279ex; height:2.509ex;" alt="{\displaystyle f}" loading="lazy"> is the <a href="Probability_density_function" title="Probability density function">probability density function</a> of a multivariate normal.

In the last equality, for each i, one indicator <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbb {I} (z_{i}=j)}">

</mrow>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbb {I} (z_{i}=j)}</annotation>

</semantics>

</math><img src="./7554fde3a46aa69efabd1e64538c9e5de68c36f7.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.651ex; height:2.843ex;" alt="{\displaystyle \mathbb {I} (z_{i}=j)}" loading="lazy"> is equal to zero, and one indicator is equal to one. The inner sum thus reduces to one term.

Given our current estimate of the parameters θ(t), the conditional distribution of the Zi is determined by <a href="Bayes'_theorem" title="Bayes' theorem">Bayes' theorem</a> to be the proportional height of the normal <a href="Probability_density_function" title="Probability density function">density</a> weighted by τ:

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<mfrac>

<mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

<mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T_{j,i}^{(t)}:=\operatorname {P} (Z_{i}=j\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})={\frac {\tau _{j}^{(t)}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{j}^{(t)},\Sigma _{j}^{(t)})}{\tau _{1}^{(t)}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{1}^{(t)},\Sigma _{1}^{(t)})+\tau _{2}^{(t)}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{2}^{(t)},\Sigma _{2}^{(t)})}}.}</annotation>

</semantics>

</math><img src="./cbd5cf1ea52696745ea83c2e67493e9e41d8b67f.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.505ex; width:76.717ex; height:8.509ex;" alt="{\displaystyle T_{j,i}^{(t)}:=\operatorname {P} (Z_{i}=j\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})={\frac {\tau _{j}^{(t)}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{j}^{(t)},\Sigma _{j}^{(t)})}{\tau _{1}^{(t)}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{1}^{(t)},\Sigma _{1}^{(t)})+\tau _{2}^{(t)}\ f(\mathbf {x} _{i};{\boldsymbol {\mu }}_{2}^{(t)},\Sigma _{2}^{(t)})}}.}" loading="lazy"></dd></dl>

These are called the "membership probabilities", which are normally considered the output of the E step (although this is not the Q function of below).

This E step corresponds with setting up this function for Q:

<mtr>

<mtd>

<msup>

</mrow>

</msup>

</mtd>

<mtd>

<msub>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mrow>

</mrow>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

<msub>

</mrow>

</mrow>

</mrow>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

<msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msup>

</mrow>

</mrow>

</msup>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

<msub>

</mrow>

</msub>

<mfrac>

</mfrac>

</mstyle>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

<mfrac>

</mfrac>

</mstyle>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<mfrac>

</mfrac>

</mstyle>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}Q(\theta \mid \theta ^{(t)})&=\operatorname {E} _{\mathbf {Z} \mid \mathbf {X} =\mathbf {x} ;\mathbf {\theta } ^{(t)}}[\log L(\theta ;\mathbf {x} ,\mathbf {Z} )]\\&=\operatorname {E} _{\mathbf {Z} \mid \mathbf {X} =\mathbf {x} ;\mathbf {\theta } ^{(t)}}[\log \prod _{i=1}^{n}L(\theta ;\mathbf {x} _{i},Z_{i})]\\&=\operatorname {E} _{\mathbf {Z} \mid \mathbf {X} =\mathbf {x} ;\mathbf {\theta } ^{(t)}}[\sum _{i=1}^{n}\log L(\theta ;\mathbf {x} _{i},Z_{i})]\\&=\sum _{i=1}^{n}\operatorname {E} _{Z_{i}\mid X_{i}=x_{i};\mathbf {\theta } ^{(t)}}[\log L(\theta ;\mathbf {x} _{i},Z_{i})]\\&=\sum _{i=1}^{n}\sum _{j=1}^{2}P(Z_{i}=j\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})\log L(\theta _{j};\mathbf {x} _{i},j)\\&=\sum _{i=1}^{n}\sum _{j=1}^{2}T_{j,i}^{(t)}{\big [}\log \tau _{j}-{\tfrac {1}{2}}\log |\Sigma _{j}|-{\tfrac {1}{2}}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})^{\top }\Sigma _{j}^{-1}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})-{\tfrac {d}{2}}\log(2\pi ){\big ]}.\end{aligned}}}</annotation>

</semantics>

</math><img src="./c4c12cc7f548a84bb750a77d2faf71cb6f06dfd4.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -19.838ex; width:86.465ex; height:40.843ex;" alt="{\displaystyle {\begin{aligned}Q(\theta \mid \theta ^{(t)})&=\operatorname {E} _{\mathbf {Z} \mid \mathbf {X} =\mathbf {x} ;\mathbf {\theta } ^{(t)}}[\log L(\theta ;\mathbf {x} ,\mathbf {Z} )]\\&=\operatorname {E} _{\mathbf {Z} \mid \mathbf {X} =\mathbf {x} ;\mathbf {\theta } ^{(t)}}[\log \prod _{i=1}^{n}L(\theta ;\mathbf {x} _{i},Z_{i})]\\&=\operatorname {E} _{\mathbf {Z} \mid \mathbf {X} =\mathbf {x} ;\mathbf {\theta } ^{(t)}}[\sum _{i=1}^{n}\log L(\theta ;\mathbf {x} _{i},Z_{i})]\\&=\sum _{i=1}^{n}\operatorname {E} _{Z_{i}\mid X_{i}=x_{i};\mathbf {\theta } ^{(t)}}[\log L(\theta ;\mathbf {x} _{i},Z_{i})]\\&=\sum _{i=1}^{n}\sum _{j=1}^{2}P(Z_{i}=j\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})\log L(\theta _{j};\mathbf {x} _{i},j)\\&=\sum _{i=1}^{n}\sum _{j=1}^{2}T_{j,i}^{(t)}{\big [}\log \tau _{j}-{\tfrac {1}{2}}\log |\Sigma _{j}|-{\tfrac {1}{2}}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})^{\top }\Sigma _{j}^{-1}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{j})-{\tfrac {d}{2}}\log(2\pi ){\big ]}.\end{aligned}}}" loading="lazy"></dd></dl>

The expectation of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \log L(\theta ;\mathbf {x} _{i},Z_{i})}">

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \log L(\theta ;\mathbf {x} _{i},Z_{i})}</annotation>

</semantics>

</math><img src="./c9d7c15b71808c0aa9700ac4abbea1ac788071e6.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:14.507ex; height:2.843ex;" alt="{\displaystyle \log L(\theta ;\mathbf {x} _{i},Z_{i})}" loading="lazy"> inside the sum is taken with respect to the probability density function <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle P(Z_{i}\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle P(Z_{i}\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})}</annotation>

</semantics>

</math><img src="./a6cb63d21c4798790355502371133afb825970d3.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:20.142ex; height:3.343ex;" alt="{\displaystyle P(Z_{i}\mid X_{i}=\mathbf {x} _{i};\theta ^{(t)})}" loading="lazy">, which might be different for each <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \mathbf {x} _{i}}">

<msub>

</mrow>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \mathbf {x} _{i}}</annotation>

</semantics>

</math><img src="./57d2ef3df60acdb53bdf90535264041fea7231cd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:2.211ex; height:2.009ex;" alt="{\displaystyle \mathbf {x} _{i}}" loading="lazy"> of the training set. Everything in the E step is known before the step is taken except <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle T_{j,i}}">

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle T_{j,i}}</annotation>

</semantics>

</math><img src="./896bd3433b93d00916b90aa6493c84434596c232.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.005ex; width:3.292ex; height:2.843ex;" alt="{\displaystyle T_{j,i}}" loading="lazy">, which is computed according to the equation at the beginning of the E step section.

This full conditional expectation does not need to be calculated in one step, because τ and μ/Σ appear in separate linear terms and can thus be maximized independently.

<msup>

</mrow>

</msup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle Q(\theta \mid \theta ^{(t)})}</annotation>

</semantics>

</math><img src="./c8b54e65c55fb22de728162c5e1ddd7314976e9b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:9.871ex; height:3.343ex;" alt="{\displaystyle Q(\theta \mid \theta ^{(t)})}" loading="lazy"> being quadratic in form means that determining the maximizing values of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \theta }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \theta }</annotation>

</semantics>

</math><img src="./6e5ab2664b422d53eb0c7df3b87e1360d75ad9af.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.09ex; height:2.176ex;" alt="{\displaystyle \theta }" loading="lazy"> is relatively straightforward. Also, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau }</annotation>

</semantics>

</math><img src="./38a7dcde9730ef0853809fefc18d88771f95206c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.202ex; height:1.676ex;" alt="{\displaystyle \tau }" loading="lazy">, <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle ({\boldsymbol {\mu }}_{1},\Sigma _{1})}">

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle ({\boldsymbol {\mu }}_{1},\Sigma _{1})}</annotation>

</semantics>

</math><img src="./3364e5b9ac90ca67273d7d3debc0a2e9c0ae125a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.275ex; height:2.843ex;" alt="{\displaystyle ({\boldsymbol {\mu }}_{1},\Sigma _{1})}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle ({\boldsymbol {\mu }}_{2},\Sigma _{2})}">

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle ({\boldsymbol {\mu }}_{2},\Sigma _{2})}</annotation>

</semantics>

</math><img src="./f921e9edd14dec88d45fcbafa7981ddb274b2958.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.275ex; height:2.843ex;" alt="{\displaystyle ({\boldsymbol {\mu }}_{2},\Sigma _{2})}" loading="lazy"> may all be maximized independently since they all appear in separate linear terms.

To begin, consider <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau }</annotation>

</semantics>

</math><img src="./38a7dcde9730ef0853809fefc18d88771f95206c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.202ex; height:1.676ex;" alt="{\displaystyle \tau }" loading="lazy">, which has the constraint <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \tau _{1}+\tau _{2}=1}">

<msub>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau _{1}+\tau _{2}=1}</annotation>

</semantics>

</math><img src="./56c54a77e3b770e7602caa4fffd1a4282fbc4684.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.671ex; width:11.242ex; height:2.509ex;" alt="{\displaystyle \tau _{1}+\tau _{2}=1}" loading="lazy">:

<mtr>

<mtd>

<msup>

</mrow>

</mrow>

</msup>

</mtd>

<mtd>

</mrow>

</munder>

</mrow>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

</munder>

</mrow>

<mrow>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

<mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}{\boldsymbol {\tau }}^{(t+1)}&={\underset {\boldsymbol {\tau }}{\operatorname {arg\,max} }}\ Q(\theta \mid \theta ^{(t)})\\&={\underset {\boldsymbol {\tau }}{\operatorname {arg\,max} }}\ \left\{\left[\sum _{i=1}^{n}T_{1,i}^{(t)}\right]\log \tau _{1}+\left[\sum _{i=1}^{n}T_{2,i}^{(t)}\right]\log \tau _{2}\right\}.\end{aligned}}}</annotation>

</semantics>

</math><img src="./f3d2b6d3ae054fad990836865c545ef1d4f1fffd.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -5.671ex; width:58.262ex; height:12.509ex;" alt="{\displaystyle {\begin{aligned}{\boldsymbol {\tau }}^{(t+1)}&={\underset {\boldsymbol {\tau }}{\operatorname {arg\,max} }}\ Q(\theta \mid \theta ^{(t)})\\&={\underset {\boldsymbol {\tau }}{\operatorname {arg\,max} }}\ \left\{\left[\sum _{i=1}^{n}T_{1,i}^{(t)}\right]\log \tau _{1}+\left[\sum _{i=1}^{n}T_{2,i}^{(t)}\right]\log \tau _{2}\right\}.\end{aligned}}}" loading="lazy"></dd></dl>

This has the same form as the maximum likelihood estimate for the <a href="Binomial_distribution" title="Binomial distribution">binomial distribution</a>, so

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

<mfrac>

</mfrac>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \tau _{j}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{j,i}^{(t)}}{\sum _{i=1}^{n}(T_{1,i}^{(t)}+T_{2,i}^{(t)})}}={\frac {1}{n}}\sum _{i=1}^{n}T_{j,i}^{(t)}.}</annotation>

</semantics>

</math><img src="./d2f22acb3fec62d449d1d6651beb6f5ba048d33c.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.838ex; width:40.983ex; height:8.843ex;" alt="{\displaystyle \tau _{j}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{j,i}^{(t)}}{\sum _{i=1}^{n}(T_{1,i}^{(t)}+T_{2,i}^{(t)})}}={\frac {1}{n}}\sum _{i=1}^{n}T_{j,i}^{(t)}.}" loading="lazy"></dd></dl>

For the next estimates of <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle ({\boldsymbol {\mu }}_{1},\Sigma _{1})}">

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle ({\boldsymbol {\mu }}_{1},\Sigma _{1})}</annotation>

</semantics>

</math><img src="./3364e5b9ac90ca67273d7d3debc0a2e9c0ae125a.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.838ex; width:8.275ex; height:2.843ex;" alt="{\displaystyle ({\boldsymbol {\mu }}_{1},\Sigma _{1})}" loading="lazy">:

<mtr>

<mtd>

</mrow>

</mrow>

</mrow>

</msubsup>

</mrow>

</mrow>

</msubsup>

</mtd>

<mtd>

</mrow>

<mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</munder>

</mrow>

<msup>

</mrow>

</msup>

</mtd>

</mtr>

<mtr>

<mtd>

</mrow>

<mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</msub>

</mrow>

</munder>

</mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<mrow>

<mrow>

<mfrac>

</mfrac>

</mstyle>

</mrow>

</mrow>

<msub>

</mrow>

</msub>

</mrow>

<mfrac>

</mfrac>

</mstyle>

</mrow>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mtd>

</mtr>

</mtable>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\begin{aligned}({\boldsymbol {\mu }}_{1}^{(t+1)},\Sigma _{1}^{(t+1)})&={\underset {{\boldsymbol {\mu }}_{1},\Sigma _{1}}{\operatorname {arg\,max} }}\ Q(\theta \mid \theta ^{(t)})\\&={\underset {{\boldsymbol {\mu }}_{1},\Sigma _{1}}{\operatorname {arg\,max} }}\ \sum _{i=1}^{n}T_{1,i}^{(t)}\left\{-{\tfrac {1}{2}}\log |\Sigma _{1}|-{\tfrac {1}{2}}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1})^{\top }\Sigma _{1}^{-1}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1})\right\}\end{aligned}}.}</annotation>

</semantics>

</math><img src="./e395f0bf406a91c497ad82b273c9d6b86759e394.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -5.838ex; width:79.036ex; height:12.843ex;" alt="{\displaystyle {\begin{aligned}({\boldsymbol {\mu }}_{1}^{(t+1)},\Sigma _{1}^{(t+1)})&={\underset {{\boldsymbol {\mu }}_{1},\Sigma _{1}}{\operatorname {arg\,max} }}\ Q(\theta \mid \theta ^{(t)})\\&={\underset {{\boldsymbol {\mu }}_{1},\Sigma _{1}}{\operatorname {arg\,max} }}\ \sum _{i=1}^{n}T_{1,i}^{(t)}\left\{-{\tfrac {1}{2}}\log |\Sigma _{1}|-{\tfrac {1}{2}}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1})^{\top }\Sigma _{1}^{-1}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1})\right\}\end{aligned}}.}" loading="lazy"></dd></dl>

This has the same form as a weighted maximum likelihood estimate for a normal distribution, so

</mrow>

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\mu }}_{1}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{1,i}^{(t)}\mathbf {x} _{i}}{\sum _{i=1}^{n}T_{1,i}^{(t)}}}}</annotation>

</semantics>

</math><img src="./45f3e73f50d396aadc98182709eee0c0d513aa6b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.838ex; width:21.563ex; height:8.843ex;" alt="{\displaystyle {\boldsymbol {\mu }}_{1}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{1,i}^{(t)}\mathbf {x} _{i}}{\sum _{i=1}^{n}T_{1,i}^{(t)}}}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Sigma _{1}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{1,i}^{(t)}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1}^{(t+1)})(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1}^{(t+1)})^{\top }}{\sum _{i=1}^{n}T_{1,i}^{(t)}}}}">

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Sigma _{1}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{1,i}^{(t)}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1}^{(t+1)})(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1}^{(t+1)})^{\top }}{\sum _{i=1}^{n}T_{1,i}^{(t)}}}}</annotation>

</semantics>

</math><img src="./a92651be432155520db19dc0b4da807039d96eb0.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.838ex; width:46.319ex; height:8.843ex;" alt="{\displaystyle \Sigma _{1}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{1,i}^{(t)}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1}^{(t+1)})(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{1}^{(t+1)})^{\top }}{\sum _{i=1}^{n}T_{1,i}^{(t)}}}}" loading="lazy"></dd></dl>

and, by symmetry,

</mrow>

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle {\boldsymbol {\mu }}_{2}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{2,i}^{(t)}\mathbf {x} _{i}}{\sum _{i=1}^{n}T_{2,i}^{(t)}}}}</annotation>

</semantics>

</math><img src="./939349990d3a58d89689a8872f23fbd27c828a43.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.838ex; width:21.563ex; height:8.843ex;" alt="{\displaystyle {\boldsymbol {\mu }}_{2}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{2,i}^{(t)}\mathbf {x} _{i}}{\sum _{i=1}^{n}T_{2,i}^{(t)}}}}" loading="lazy"> and <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \Sigma _{2}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{2,i}^{(t)}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{2}^{(t+1)})(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{2}^{(t+1)})^{\top }}{\sum _{i=1}^{n}T_{2,i}^{(t)}}}.}">

</mrow>

</mrow>

</msubsup>

<mfrac>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

<msub>

</mrow>

</mrow>

</msub>

</mrow>

</mrow>

</mrow>

</msubsup>

<msup>

</mrow>

</msup>

</mrow>

<mrow>

</mrow>

</mrow>

</munderover>

</mrow>

</mrow>

</msubsup>

</mrow>

</mfrac>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \Sigma _{2}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{2,i}^{(t)}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{2}^{(t+1)})(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{2}^{(t+1)})^{\top }}{\sum _{i=1}^{n}T_{2,i}^{(t)}}}.}</annotation>

</semantics>

</math><img src="./88bd69bf8025781f8cad373b62fbdc322e5c2b0e.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -3.838ex; width:46.966ex; height:8.843ex;" alt="{\displaystyle \Sigma _{2}^{(t+1)}={\frac {\sum _{i=1}^{n}T_{2,i}^{(t)}(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{2}^{(t+1)})(\mathbf {x} _{i}-{\boldsymbol {\mu }}_{2}^{(t+1)})^{\top }}{\sum _{i=1}^{n}T_{2,i}^{(t)}}}.}" loading="lazy"></dd></dl>

<div class="mw-heading mw-heading4"><h4 id="Termination">Termination</h4></div>

Conclude the iterative process if <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle E_{Z\mid \theta ^{(t)},\mathbf {x} }[\log L(\theta ^{(t)};\mathbf {x} ,\mathbf {Z} )]\leq E_{Z\mid \theta ^{(t-1)},\mathbf {x} }[\log L(\theta ^{(t-1)};\mathbf {x} ,\mathbf {Z} )]+\varepsilon }">

<msub>

<msup>

</mrow>

</msup>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

<msub>

<msup>

</mrow>

</msup>

</mrow>

</msub>

<msup>

</mrow>

</msup>

</mrow>

</mrow>

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle E_{Z\mid \theta ^{(t)},\mathbf {x} }[\log L(\theta ^{(t)};\mathbf {x} ,\mathbf {Z} )]\leq E_{Z\mid \theta ^{(t-1)},\mathbf {x} }[\log L(\theta ^{(t-1)};\mathbf {x} ,\mathbf {Z} )]+\varepsilon }</annotation>

</semantics>

</math><img src="./1936cdfb30674d85c6029aed6c4b3e571300374b.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -1.505ex; width:58.543ex; height:4.009ex;" alt="{\displaystyle E_{Z\mid \theta ^{(t)},\mathbf {x} }[\log L(\theta ^{(t)};\mathbf {x} ,\mathbf {Z} )]\leq E_{Z\mid \theta ^{(t-1)},\mathbf {x} }[\log L(\theta ^{(t-1)};\mathbf {x} ,\mathbf {Z} )]+\varepsilon }" loading="lazy"> for <math xmlns="http://www.w3.org/1998/Math/MathML" alttext="{\displaystyle \varepsilon }">

</mstyle>

</mrow>

<annotation encoding="application/x-tex">{\displaystyle \varepsilon }</annotation>

</semantics>

</math><img src="./a30c89172e5b88edbd45d3e2772c7f5e562e5173.svg" class="mwe-math-fallback-image-inline mw-invert skin-invert" aria-hidden="true" style="vertical-align: -0.338ex; width:1.083ex; height:1.676ex;" alt="{\displaystyle \varepsilon }" loading="lazy"> below some preset threshold.

<div class="mw-heading mw-heading4"><h4 id="Generalization">Generalization</h4></div>

The algorithm illustrated above can be generalized for mixtures of more than two <a href="Multivariate_normal_distribution" title="Multivariate normal distribution">multivariate normal distributions</a>.

<div class="mw-heading mw-heading3"><h3 id="Truncated_and_censored_regression">Truncated and censored regression</h3></div>

The EM algorithm has been implemented in the case where an underlying <a href="Linear_regression" title="Linear regression">linear regression</a> model exists explaining the variation of some quantity, but where the values actually observed are censored or truncated versions of those represented in the model.<a href="#cite_note-Wolynetz-38">[38]</a> Special cases of this model include censored or truncated observations from one <a href="Normal_distribution" title="Normal distribution">normal distribution</a>.<a href="#cite_note-Wolynetz-38">[38]</a>

<div class="mw-heading mw-heading2"><h2 id="Alternatives">Alternatives</h2></div>

EM typically converges to a local optimum, not necessarily the global optimum, with no bound on the convergence rate in general. It is possible that it can be arbitrarily poor in high dimensions and there can be an exponential number of local optima. Hence, a need exists for alternative methods for guaranteed learning, especially in the high-dimensional setting. Alternatives to EM exist with better guarantees for consistency, which are termed moment-based approaches<a href="#cite_note-39">[39]</a> or the so-called spectral techniques.<a href="#cite_note-40">[40]</a><a href="#cite_note-41">[41]</a> Moment-based approaches to learning the parameters of a probabilistic model enjoy guarantees such as global convergence under certain conditions unlike EM which is often plagued by the issue of getting stuck in local optima. Algorithms with guarantees for learning can be derived for a number of important models such as mixture models, HMMs etc. For these spectral methods, no spurious local optima occur, and the true parameters can be consistently estimated under some regularity conditions.

<ul><li><a href="Mixture_distribution" title="Mixture distribution">mixture distribution</a></li>

<li><a href="Compound_distribution" class="mw-redirect" title="Compound distribution">compound distribution</a></li>

<li><a href="Density_estimation" title="Density estimation">density estimation</a></li>

<li><a href="Principal_component_analysis" title="Principal component analysis">Principal component analysis</a></li>

<li><a href="Total_absorption_spectroscopy" title="Total absorption spectroscopy">total absorption spectroscopy</a></li>

<li>The EM algorithm can be viewed as a special case of the <a href="MM_algorithm" title="MM algorithm">majorize-minimization (MM) algorithm</a>.<a href="#cite_note-42">[42]</a></li></ul>

<div class="mw-heading mw-heading2"><h2 id="References">References</h2></div>

/* start https://en.wikipedia.org/ */

.mw-parser-output .reflist{margin-bottom:0.5em;list-style-type:decimal}@media screen{.mw-parser-output .reflist{font-size:90%}}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}

/* end https://en.wikipedia.org/ */

</style><div class="reflist">

/* start https://en.wikipedia.org/ */

.mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free.id-lock-free a{background:url("./mw/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited.id-lock-limited a,.mw-parser-output .id-lock-registration.id-lock-registration a{background:url("./mw/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription.id-lock-subscription a{background:url("./mw/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("./mw/Wikisource-logo.svg")right 0.1em center/12px no-repeat}body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-free a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-limited a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-registration a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .id-lock-subscription a,body:not(.skin-timeless):not(.skin-minerva) .mw-parser-output .cs1-ws-icon a{background-size:contain;padding:0 1em 0 0}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:var(--color-error,#d33)}.mw-parser-output .cs1-visible-error{color:var(--color-error,#d33)}.mw-parser-output .cs1-maint{display:none;color:#085;margin-left:0.3em}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}@media screen{.mw-parser-output .cs1-format{font-size:95%}html.skin-theme-clientpref-night .mw-parser-output .cs1-maint{color:#18911f}}@media screen and (prefers-color-scheme:dark){html.skin-theme-clientpref-os .mw-parser-output .cs1-maint{color:#18911f}}

/* end https://en.wikipedia.org/ */

</style><cite id="CITEREFMengvan_Dyk1997" class="citation journal cs1">Meng, X.-L.; van Dyk, D. (1997). <a rel="nofollow" class="external text" href="https://doi.org/10.1111%2F1467-9868.00082">"The EM algorithm – an old folk-song sung to a fast new tune"</a>. J. Royal Statist. Soc. B. 59 (3): 511–567. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2F1467-9868.00082">10.1111/1467-9868.00082</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:17461647">17461647</a>.</cite>

</li>

<li id="cite_note-2"><a href="#cite_ref-2">^</a> Jeongyeol Kwon, Constantine Caramanis

Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:1727-1736, 2020.

</li>

<cite id="CITEREFDempsterLairdRubin1977" class="citation journal cs1"><a href="Arthur_P._Dempster" title="Arthur P. Dempster">Dempster, A.P.</a>; <a href="Nan_Laird" title="Nan Laird">Laird, N.M.</a>; <a href="Donald_Rubin" title="Donald Rubin">Rubin, D.B.</a> (1977). "Maximum Likelihood from Incomplete Data via the EM Algorithm". <a href="Journal_of_the_Royal_Statistical_Society%2C_Series_B" class="mw-redirect" title="Journal of the Royal Statistical Society, Series B">Journal of the Royal Statistical Society, Series B</a>. 39 (1): 1–38. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Fj.2517-6161.1977.tb01600.x">10.1111/j.2517-6161.1977.tb01600.x</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2984875">2984875</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=0501537">0501537</a>.</cite>

</li>

<li id="cite_note-4"><a href="#cite_ref-4">^</a> <cite id="CITEREFCeppelini1955" class="citation journal cs1">Ceppelini, R.M. (1955). "The estimation of gene frequencies in a random-mating population". Ann. Hum. Genet. 20 (2): 97–115. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Fj.1469-1809.1955.tb01360.x">10.1111/j.1469-1809.1955.tb01360.x</a>. <a href="PMID_(identifier)" class="mw-redirect" title="PMID (identifier)">PMID</a> <a rel="nofollow" class="external text" href="https://pubmed.ncbi.nlm.nih.gov/13268982">13268982</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:38625779">38625779</a>.</cite>

</li>

<li id="cite_note-5"><a href="#cite_ref-5">^</a> <cite id="CITEREFHartley1958" class="citation journal cs1">Hartley, Herman Otto (1958). "Maximum Likelihood estimation from incomplete data". Biometrics. 14 (2): 174–194. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2307%2F2527783">10.2307/2527783</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2527783">2527783</a>.</cite>

</li>

<li id="cite_note-6"><a href="#cite_ref-6">^</a> <cite id="CITEREFNgKrishnanMcLachlan2011" class="citation cs2">Ng, Shu Kay; Krishnan, Thriyambakam; McLachlan, Geoffrey J. (2011-12-21), <a rel="nofollow" class="external text" href="https://dx.doi.org/10.1007/978-3-642-21551-3_6">"The EM Algorithm"</a>, Handbook of Computational Statistics, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 139–172, <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1007%2F978-3-642-21551-3_6">10.1007/978-3-642-21551-3_6</a>, <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-3-642-21550-6</bdi>, <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:59942212">59942212</a>, retrieved 2022-10-15</cite>

</li>

<li id="cite_note-Sundberg1974-7"><a href="#cite_ref-Sundberg1974_7-0">^</a> <cite id="CITEREFSundberg1974" class="citation journal cs1">Sundberg, Rolf (1974). "Maximum likelihood theory for incomplete data from an exponential family". Scandinavian Journal of Statistics. 1 (2): 49–58. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/4615553">4615553</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=0381110">0381110</a>.</cite>

</li>

Rolf Sundberg. 1971. Maximum likelihood theory and applications for distributions generated when observing a function of an exponential family variable. Dissertation, Institute for Mathematical Statistics, Stockholm University.

</li>

<li id="cite_note-Sundberg1976-9">^ <a href="#cite_ref-Sundberg1976_9-0">a</a> <a href="#cite_ref-Sundberg1976_9-1">b</a> <cite id="CITEREFSundberg1976" class="citation journal cs1">Sundberg, Rolf (1976). "An iterative method for solution of the likelihood equations for incomplete data from exponential families". Communications in Statistics – Simulation and Computation. 5 (1): 55–64. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1080%2F03610917608812007">10.1080/03610917608812007</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=0443190">0443190</a>.</cite>

</li>

<li id="cite_note-10"><a href="#cite_ref-10">^</a> See the acknowledgement by Dempster, Laird and Rubin on pages 3, 5 and 11.

</li>

<li id="cite_note-Martin-Löf1966-11">^ <a href="#cite_ref-Martin-Löf1966_11-0">a</a> <a href="#cite_ref-Martin-Löf1966_11-1">b</a> <a href="Per_Martin-L%C3%B6f" title="Per Martin-Löf">Per Martin-Löf</a>. 1966. Statistics from the point of view of statistical mechanics. Lecture notes, Mathematical Institute, Aarhus University. ("Sundberg formula", credited to Anders Martin-Löf).

</li>

<li id="cite_note-Martin-Löf1970-12">^ <a href="#cite_ref-Martin-Löf1970_12-0">a</a> <a href="#cite_ref-Martin-Löf1970_12-1">b</a> <a href="Per_Martin-L%C3%B6f" title="Per Martin-Löf">Per Martin-Löf</a>. 1970. Statistiska Modeller (Statistical Models): Anteckningar från seminarier läsåret 1969–1970 (Lecture notes 1969-1970), with the assistance of Rolf Sundberg. Stockholm University.

</li>

<li id="cite_note-Martin-Löf1974a-13">^ <a href="#cite_ref-Martin-Löf1974a_13-0">a</a> <a href="#cite_ref-Martin-Löf1974a_13-1">b</a> Martin-Löf, P. The notion of redundancy and its use as a quantitative measure of the deviation between a statistical hypothesis and a set of observational data. With a discussion by F. Abildgård, <a href="Arthur_P._Dempster" title="Arthur P. Dempster">A. P. Dempster</a>, <a href="D._Basu" class="mw-redirect" title="D. Basu">D. Basu</a>, <a href="D._R._Cox" class="mw-redirect" title="D. R. Cox">D. R. Cox</a>, <a href="A._W._F._Edwards" title="A. W. F. Edwards">A. W. F. Edwards</a>, D. A. Sprott, <a href="George_A._Barnard" class="mw-redirect" title="George A. Barnard">G. A. Barnard</a>, O. Barndorff-Nielsen, J. D. Kalbfleisch and <a href="Rasch_model" title="Rasch model">G. Rasch</a> and a reply by the author. Proceedings of Conference on Foundational Questions in Statistical Inference (Aarhus, 1973), pp. 1–42. Memoirs, No. 1, Dept. Theoret. Statist., Inst. Math., Univ. Aarhus, Aarhus, 1974.

</li>

<li id="cite_note-Martin-Löf1974b-14">^ <a href="#cite_ref-Martin-Löf1974b_14-0">a</a> <a href="#cite_ref-Martin-Löf1974b_14-1">b</a> <cite id="CITEREFMartin-Löf1974" class="citation journal cs1">Martin-Löf, Per (1974). "The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and a set of observational data". Scand. J. Statist. 1 (1): 3–18.</cite>

</li>

<cite id="CITEREFWu1983" class="citation journal cs1">Wu, C. F. Jeff (Mar 1983). <a rel="nofollow" class="external text" href="https://doi.org/10.1214%2Faos%2F1176346060">"On the Convergence Properties of the EM Algorithm"</a>. <a href="Annals_of_Statistics" title="Annals of Statistics">Annals of Statistics</a>. 11 (1): 95–103. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1214%2Faos%2F1176346060">10.1214/aos/1176346060</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2240463">2240463</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=0684867">0684867</a>.</cite>

</li>

<li id="cite_note-16"><a href="#cite_ref-16">^</a> <cite id="CITEREFSundberg2019" class="citation book cs1">Sundberg, Rolf (2019). Statistical Modelling by Exponential Families. Cambridge University Press. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>9781108701112</bdi>.</cite>

</li>

<li id="cite_note-17"><a href="#cite_ref-17">^</a> <cite id="CITEREFLaird2006" class="citation book cs1">Laird, Nan (2006). <a rel="nofollow" class="external text" href="https://doi.org/10.1002/0471667196.ess2643.pub2">"Sundberg formulas"</a>. Encyclopedia of Statistical Sciences. Wiley. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1002%2F0471667196.ess2643.pub2">10.1002/0471667196.ess2643.pub2</a>. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>0471667196</bdi>.</cite>

</li>

<li id="cite_note-Little1987-18"><a href="#cite_ref-Little1987_18-0">^</a> <cite id="CITEREFLittleRubin1987" class="citation book cs1">Little, Roderick J.A.; <a href="Donald_Rubin" title="Donald Rubin">Rubin, Donald B.</a> (1987). <a rel="nofollow" class="external text" href="https://archive.org/details/statisticalanaly00litt">Statistical Analysis with Missing Data</a>. Wiley Series in Probability and Mathematical Statistics. New York: John Wiley & Sons. pp. <a rel="nofollow" class="external text" href="https://archive.org/details/statisticalanaly00litt/page/n145">134</a>–136. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-80254-9</bdi>.</cite>

</li>

<li id="cite_note-neal1999-19">^ <a href="#cite_ref-neal1999_19-0">a</a> <a href="#cite_ref-neal1999_19-1">b</a> <cite id="CITEREFNealHinton1999" class="citation book cs1">Neal, Radford; <a href="Geoffrey_Hinton" title="Geoffrey Hinton">Hinton, Geoffrey</a> (1999). "A view of the EM algorithm that justifies incremental, sparse, and other variants". In <a href="Michael_I._Jordan" title="Michael I. Jordan">Michael I. Jordan</a> (ed.). <a rel="nofollow" class="external text" href="http://ftp.cs.toronto.edu/pub/radford/emk.pdf">Learning in Graphical Models</a> (PDF). Cambridge, MA: MIT Press. pp. 355–368. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-262-60032-3</bdi>. Retrieved 2009-03-22.</cite>

</li>

<li id="cite_note-hastie2001-20">^ <a href="#cite_ref-hastie2001_20-0">a</a> <a href="#cite_ref-hastie2001_20-1">b</a> <cite id="CITEREFHastieTibshiraniFriedman2001" class="citation book cs1"><a href="Trevor_Hastie" title="Trevor Hastie">Hastie, Trevor</a>; <a href="Robert_Tibshirani" title="Robert Tibshirani">Tibshirani, Robert</a>; Friedman, Jerome (2001). "8.5 The EM algorithm". <a rel="nofollow" class="external text" href="https://archive.org/details/elementsstatisti00thas_842">The Elements of Statistical Learning</a>. New York: Springer. pp. <a rel="nofollow" class="external text" href="https://archive.org/details/elementsstatisti00thas_842/page/n237">236</a>–243. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-387-95284-0</bdi>.</cite>

</li>

<li id="cite_note-21"><a href="#cite_ref-21">^</a> <cite id="CITEREFLindstromBates1988" class="citation journal cs1">Lindstrom, Mary J; Bates, Douglas M (1988). "Newton—Raphson and EM Algorithms for Linear Mixed-Effects Models for Repeated-Measures Data". Journal of the American Statistical Association. 83 (404): 1014. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1080%2F01621459.1988.10478693">10.1080/01621459.1988.10478693</a>.</cite>

</li>

<li id="cite_note-22"><a href="#cite_ref-22">^</a> <cite id="CITEREFVan_Dyk2000" class="citation journal cs1">Van Dyk, David A (2000). "Fitting Mixed-Effects Models Using Efficient EM-Type Algorithms". Journal of Computational and Graphical Statistics. 9 (1): 78–98. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2307%2F1390614">10.2307/1390614</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/1390614">1390614</a>.</cite>

</li>

<li id="cite_note-23"><a href="#cite_ref-23">^</a> <cite id="CITEREFDiffeySmithWelshCullis2017" class="citation journal cs1">Diffey, S. M; Smith, A. B; Welsh, A. H; Cullis, B. R (2017). <a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Fanzs.12208">"A new REML (parameter expanded) EM algorithm for linear mixed models"</a>. Australian & New Zealand Journal of Statistics. 59 (4): 433. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2Fanzs.12208">10.1111/anzs.12208</a>. <a href="Hdl_(identifier)" class="mw-redirect" title="Hdl (identifier)">hdl</a>:<a rel="nofollow" class="external text" href="https://hdl.handle.net/1885%2F211365">1885/211365</a>.</cite>

</li>

<li id="cite_note-24"><a href="#cite_ref-24">^</a> Matarazzo, T. J., and Pakzad, S. N. (2016). “STRIDE for Structural Identification using Expectation Maximization: Iterative Output-Only Method for Modal Identification.” Journal of Engineering Mechanics.<a rel="nofollow" class="external free" href="http://ascelibrary.org/doi/abs/10.1061/(ASCE)EM.1943-7889.0000951">http://ascelibrary.org/doi/abs/10.1061/(ASCE)EM.1943-7889.0000951</a>

</li>

<li id="cite_note-25"><a href="#cite_ref-25">^</a> <cite id="CITEREFKreerKizilersuThomas2022" class="citation journal cs1">Kreer, Markus; Kizilersu, Ayse; Thomas, Anthony W. (2022). <a rel="nofollow" class="external text" href="https://www.sciencedirect.com/science/article/pii/S0378437121007299">"Censored expectation maximization algorithm for mixtures: Application to intertrade waiting times"</a>. Physica A: Statistical Mechanics and Its Applications. 587 (1): 126456. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2022PhyA..58726456K">2022PhyA..58726456K</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1016%2Fj.physa.2021.126456">10.1016/j.physa.2021.126456</a>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/0378-4371">0378-4371</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:244198364">244198364</a>.</cite>

</li>

<li id="cite_note-26"><a href="#cite_ref-26">^</a> <cite id="CITEREFEinickeMalosReidHainsworth2009" class="citation journal cs1">Einicke, G. A.; Malos, J. T.; Reid, D. C.; Hainsworth, D. W. (January 2009). "Riccati Equation and EM Algorithm Convergence for Inertial Navigation Alignment". IEEE Trans. Signal Process. 57 (1): 370–375. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2009ITSP...57..370E">2009ITSP...57..370E</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTSP.2008.2007090">10.1109/TSP.2008.2007090</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:1930004">1930004</a>.</cite>

</li>

<li id="cite_note-27"><a href="#cite_ref-27">^</a> <cite id="CITEREFEinickeFalcoMalos2010" class="citation journal cs1">Einicke, G. A.; Falco, G.; Malos, J. T. (May 2010). "EM Algorithm State Matrix Estimation for Navigation". IEEE Signal Processing Letters. 17 (5): 437–440. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2010ISPL...17..437E">2010ISPL...17..437E</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FLSP.2010.2043151">10.1109/LSP.2010.2043151</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:14114266">14114266</a>.</cite>

</li>

<li id="cite_note-28"><a href="#cite_ref-28">^</a> <cite id="CITEREFEinickeFalcoDunnReid2012" class="citation journal cs1">Einicke, G. A.; Falco, G.; Dunn, M. T.; Reid, D. C. (May 2012). "Iterative Smoother-Based Variance Estimation". IEEE Signal Processing Letters. 19 (5): 275–278. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2012ISPL...19..275E">2012ISPL...19..275E</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FLSP.2012.2190278">10.1109/LSP.2012.2190278</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:17476971">17476971</a>.</cite>

</li>

<li id="cite_note-29"><a href="#cite_ref-29">^</a> <cite id="CITEREFEinicke2015" class="citation journal cs1">Einicke, G. A. (Sep 2015). "Iterative Filtering and Smoothing of Measurements Possessing Poisson Noise". IEEE Transactions on Aerospace and Electronic Systems. 51 (3): 2205–2011. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/2015ITAES..51.2205E">2015ITAES..51.2205E</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTAES.2015.140843">10.1109/TAES.2015.140843</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:32667132">32667132</a>.</cite>

</li>

<li id="cite_note-30"><a href="#cite_ref-30">^</a> <cite id="CITEREFJamshidianJennrich1997" class="citation journal cs1">Jamshidian, Mortaza; Jennrich, Robert I. (1997). "Acceleration of the EM Algorithm by using Quasi-Newton Methods". <a href="Journal_of_the_Royal_Statistical_Society%2C_Series_B" class="mw-redirect" title="Journal of the Royal Statistical Society, Series B">Journal of the Royal Statistical Society, Series B</a>. 59 (2): 569–587. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1111%2F1467-9868.00083">10.1111/1467-9868.00083</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=1452026">1452026</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:121966443">121966443</a>.</cite>

</li>

<li id="cite_note-31"><a href="#cite_ref-31">^</a> <cite id="CITEREFLiu1998" class="citation journal cs1">Liu, C (1998). "Parameter expansion to accelerate EM: The PX-EM algorithm". Biometrika. 85 (4): 755–770. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.134.9617">10.1.1.134.9617</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1093%2Fbiomet%2F85.4.755">10.1093/biomet/85.4.755</a>.</cite>

</li>

<li id="cite_note-32"><a href="#cite_ref-32">^</a> <cite id="CITEREFMengRubin1993" class="citation journal cs1">Meng, Xiao-Li; <a href="Donald_Rubin" title="Donald Rubin">Rubin, Donald B.</a> (1993). "Maximum likelihood estimation via the ECM algorithm: A general framework". <a href="Biometrika" title="Biometrika">Biometrika</a>. 80 (2): 267–278. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1093%2Fbiomet%2F80.2.267">10.1093/biomet/80.2.267</a>. <a href="MR_(identifier)" class="mw-redirect" title="MR (identifier)">MR</a> <a rel="nofollow" class="external text" href="https://mathscinet.ams.org/mathscinet-getitem?mr=1243503">1243503</a>. <a href="S2CID_(identifier)" class="mw-redirect" title="S2CID (identifier)">S2CID</a> <a rel="nofollow" class="external text" href="https://api.semanticscholar.org/CorpusID:40571416">40571416</a>.</cite>

</li>

<li id="cite_note-33"><a href="#cite_ref-33">^</a> <cite id="CITEREFLiuRubin1994" class="citation journal cs1">Liu, Chuanhai; Rubin, Donald B (1994). "The ECME Algorithm: A Simple Extension of EM and ECM with Faster Monotone Convergence". Biometrika. 81 (4): 633. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1093%2Fbiomet%2F81.4.633">10.1093/biomet/81.4.633</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2337067">2337067</a>.</cite>

</li>

<cite id="CITEREFJiangtao_YinYanfeng_ZhangLixin_Gao2012" class="citation journal cs1">Jiangtao Yin; Yanfeng Zhang; Lixin Gao (2012). <a rel="nofollow" class="external text" href="http://rio.ecs.umass.edu/mnilpub/papers/cluster2012-yin.pdf">"Accelerating Expectation–Maximization Algorithms with Frequent Updates"</a> (PDF). Proceedings of the IEEE International Conference on Cluster Computing.</cite>

</li>

<li id="cite_note-35"><a href="#cite_ref-35">^</a> Hunter DR and Lange K (2004), <a rel="nofollow" class="external text" href="http://www.stat.psu.edu/~dhunter/papers/mmtutorial.pdf">A Tutorial on MM Algorithms</a>, The American Statistician, 58: 30–37

</li>

<cite id="CITEREFMatsuyama2003" class="citation journal cs1">Matsuyama, Yasuo (2003). "The α-EM algorithm: Surrogate likelihood maximization using α-logarithmic information measures". IEEE Transactions on Information Theory. 49 (3): 692–706. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1109%2FTIT.2002.808105">10.1109/TIT.2002.808105</a>.</cite>

</li>

<cite id="CITEREFMatsuyama2011" class="citation journal cs1">Matsuyama, Yasuo (2011). "Hidden Markov model estimation based on alpha-EM algorithm: Discrete and continuous alpha-HMMs". International Joint Conference on Neural Networks: 808–816.</cite>

</li>

<li id="cite_note-Wolynetz-38">^ <a href="#cite_ref-Wolynetz_38-0">a</a> <a href="#cite_ref-Wolynetz_38-1">b</a> <cite id="CITEREFWolynetz1979" class="citation journal cs1">Wolynetz, M.S. (1979). "Maximum likelihood estimation in a linear model from confined and censored normal data". <a href="Journal_of_the_Royal_Statistical_Society%2C_Series_C" class="mw-redirect" title="Journal of the Royal Statistical Society, Series C">Journal of the Royal Statistical Society, Series C</a>. 28 (2): 195–206. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.2307%2F2346749">10.2307/2346749</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/2346749">2346749</a>.</cite>

</li>

<li id="cite_note-39"><a href="#cite_ref-39">^</a> <cite id="CITEREFPearson1894" class="citation journal cs1">Pearson, Karl (1894). <a rel="nofollow" class="external text" href="https://doi.org/10.1098%2Frsta.1894.0003">"Contributions to the Mathematical Theory of Evolution"</a>. Philosophical Transactions of the Royal Society of London A. 185: 71–110. <a href="Bibcode_(identifier)" class="mw-redirect" title="Bibcode (identifier)">Bibcode</a>:<a rel="nofollow" class="external text" href="https://ui.adsabs.harvard.edu/abs/1894RSPTA.185...71P">1894RSPTA.185...71P</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1098%2Frsta.1894.0003">10.1098/rsta.1894.0003</a>. <a href="ISSN_(identifier)" class="mw-redirect" title="ISSN (identifier)">ISSN</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/issn/0264-3820">0264-3820</a>. <a href="JSTOR_(identifier)" class="mw-redirect" title="JSTOR (identifier)">JSTOR</a> <a rel="nofollow" class="external text" href="https://www.jstor.org/stable/90667">90667</a>.</cite>

</li>

<li id="cite_note-40"><a href="#cite_ref-40">^</a> <cite id="CITEREFShabanMehrdadBoLe2015" class="citation journal cs1">Shaban, Amirreza; Mehrdad, Farajtabar; Bo, Xie; Le, Song; Byron, Boots (2015). <a rel="nofollow" class="external text" href="https://web.archive.org/web/20161224102320/https://www.cc.gatech.edu/~bboots3/files/SpectralExteriorPoint-NIPSWorkshop.pdf">"Learning Latent Variable Models by Improving Spectral Solutions with Exterior Point Method"</a> (PDF). UAI: 792–801. Archived from <a rel="nofollow" class="external text" href="https://www.cc.gatech.edu/~bboots3/files/SpectralExteriorPoint-NIPSWorkshop.pdf">the original</a> (PDF) on 2016-12-24. Retrieved 2019-06-12.</cite>

</li>

<li id="cite_note-41"><a href="#cite_ref-41">^</a> <cite id="CITEREFBalle,_Borja_Quattoni,_Ariadna_Carreras,_Xavier2012" class="citation book cs1">Balle, Borja Quattoni, Ariadna Carreras, Xavier (2012-06-27). Local Loss Optimization in Operator Models: A New Insight into Spectral Learning. <a href="OCLC_(identifier)" class="mw-redirect" title="OCLC (identifier)">OCLC</a> <a rel="nofollow" class="external text" href="https://search.worldcat.org/oclc/815865081">815865081</a>.</cite><code class="cs1-code">{{cite book}}</code>: CS1 maint: multiple names: authors list (link)

</li>

<li id="cite_note-42"><a href="#cite_ref-42">^</a> <cite id="CITEREFLange" class="citation web cs1">Lange, Kenneth. <a rel="nofollow" class="external text" href="http://www.stat.berkeley.edu/~aldous/Colloq/lange-talk.pdf">"The MM Algorithm"</a> (PDF).</cite>

</li>

</ol></div></div>

<div class="mw-heading mw-heading2"><h2 id="Further_reading">Further reading</h2></div>

<ul><li><cite id="CITEREFHoggMcKeanCraig2005" class="citation book cs1">Hogg, Robert; McKean, Joseph; <a href="Allen_Craig" title="Allen Craig">Craig, Allen</a> (2005). Introduction to Mathematical Statistics. Upper Saddle River, NJ: Pearson Prentice Hall. pp. 359–364.</cite></li>

<li><cite id="CITEREFDellaert2002" class="citation report cs1"><a href="Frank_Dellaert" title="Frank Dellaert">Dellaert, Frank</a> (February 2002). <a rel="nofollow" class="external text" href="https://dellaert.github.io/files/em-paper.pdf">The Expectation Maximization Algorithm</a> (PDF) (Technical Report number GIT-GVU-02-20). Georgia Tech College of Computing.</cite> gives an easier explanation of EM algorithm as to lowerbound maximization.</li>

<li><cite id="CITEREFBishop2006" class="citation book cs1"><a href="Christopher_Bishop" title="Christopher Bishop">Bishop, Christopher M.</a> (2006). Pattern Recognition and Machine Learning. Springer. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-387-31073-2</bdi>.</cite></li>

<li><cite id="CITEREFGuptaChen2010" class="citation journal cs1">Gupta, M. R.; Chen, Y. (2010). "Theory and Use of the EM Algorithm". Foundations and Trends in Signal Processing. 4 (3): 223–296. <a href="CiteSeerX_(identifier)" class="mw-redirect" title="CiteSeerX (identifier)">CiteSeerX</a> <a rel="nofollow" class="external text" href="https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.219.6830">10.1.1.219.6830</a>. <a href="Doi_(identifier)" class="mw-redirect" title="Doi (identifier)">doi</a>:<a rel="nofollow" class="external text" href="https://doi.org/10.1561%2F2000000034">10.1561/2000000034</a>.</cite> A well-written short book on EM, including detailed derivation of EM for GMMs, HMMs, and Dirichlet.</li>

<li><cite id="CITEREFBilmes1997" class="citation report cs1">Bilmes, Jeff (1997). <a rel="nofollow" class="external text" href="https://people.ece.uw.edu/bilmes/p/pgs/b2hd-bilmes1997-em.html">A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models</a> (Technical Report TR-97-021). International Computer Science Institute.</cite> includes a simplified derivation of the EM equations for Gaussian Mixtures and Gaussian Mixture Hidden Markov Models.</li>

<li><cite id="CITEREFMcLachlanKrishnan2008" class="citation book cs1">McLachlan, Geoffrey J.; Krishnan, Thriyambakam (2008). The EM Algorithm and Extensions (2nd ed.). Hoboken: Wiley. <a href="ISBN_(identifier)" class="mw-redirect" title="ISBN (identifier)">ISBN</a> <bdi>978-0-471-20170-0</bdi>.</cite></li></ul>

<div class="mw-heading mw-heading2"><h2 id="External_links">External links</h2></div>

<ul><li>Various 1D, 2D and 3D <a rel="nofollow" class="external text" href="http://wiki.stat.ucla.edu/socr/index.php/SOCR_EduMaterials_Activities_2D_PointSegmentation_EM_Mixture">demonstrations of EM together with Mixture Modeling</a> are provided as part of the paired <a href="SOCR" class="mw-redirect" title="SOCR">SOCR</a> activities and applets. These applets and activities show empirically the properties of the EM algorithm for parameter estimation in diverse settings.</li>

<li><a rel="nofollow" class="external text" href="https://github.com/l-/CommonDataAnalysis">Class hierarchy</a> in <a href="C%2B%2B" title="C++">C++</a> (GPL) including Gaussian Mixtures</li>

<li><a rel="nofollow" class="external text" href="http://www.inference.phy.cam.ac.uk/mackay/itila/">The on-line textbook: Information Theory, Inference, and Learning Algorithms</a>, by <a href="David_J.C._MacKay" class="mw-redirect" title="David J.C. MacKay">David J.C. MacKay</a> includes simple examples of the EM algorithm such as clustering using the soft k-means algorithm, and emphasizes the variational view of the EM algorithm, as described in Chapter 33.7 of version 7.2 (fourth edition).</li>

<li><a rel="nofollow" class="external text" href="http://www.cse.buffalo.edu/faculty/mbeal/papers/beal03.pdf">Variational Algorithms for Approximate Bayesian Inference</a>, by M. J. Beal includes comparisons of EM to Variational Bayesian EM and derivations of several models including Variational Bayesian HMMs (<a rel="nofollow" class="external text" href="http://www.cse.buffalo.edu/faculty/mbeal/thesis/index.html">chapters</a>).</li>

<li><a rel="nofollow" class="external text" href="http://www.seanborman.com/publications/EM_algorithm.pdf">The Expectation Maximization Algorithm: A short tutorial</a>, A self-contained derivation of the EM Algorithm by Sean Borman.</li>

<li><a rel="nofollow" class="external text" href="http://pages.cs.wisc.edu/~jerryzhu/cs838/EM.pdf">The EM Algorithm</a>, by Xiaojin Zhu.</li>

<li><a rel="nofollow" class="external text" href="https://arxiv.org/abs/1105.1476">EM algorithm and variants: an informal tutorial</a> by Alexis Roche. A concise and very clear description of EM and many interesting variants.</li></ul></div><div><div class="zim-footer">

This article is issued from <a class="external text" title="Last edited on 2025-06-23" href="https://en.wikipedia.org/wiki/?title=Expectation%E2%80%93maximization_algorithm&oldid=1297006206">Wikipedia</a>. The text is available under <a class="external text" href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">Creative Commons Attribution-Share Alike 4.0</a> unless otherwise noted. Additional terms may apply for the media files.

</div>

</div></div>

</div>

</main>

</div>

</body></html>